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ABSTRACT 


Previous  work  (Ginn,  1991)  showed  that  faults  in  a  program  tend  to 
cluster  when  viewed  by  the  variables  that  affect  execution  and  propagation 
of  the  fault  (structural  clustering).  However,  that  study  was  quite  preliminary 
and  local  in  its  investigation.  This  thesis  examines  clustering  from  two  other 
perspectives,  taxonomical  (type  of  faults)  and  functional  (area  of  affected 
functionality).  The  hypothesis  tested  was  that  faults  tend  to  cluster  when 
viewed  from  these  perspectives. 

The  approach  was  to  use  chi-square  statistics  on  data  taken  from 
(Shimeall,  1991)  to  test  the  hypotheses,  247  faults  were  analyzed  and  the 
resultant  clustering  was  cross-examined  across  the  perspectives. 

The  results  show  that  the  studied  failure  regions  have  a  strong  tendency 
to  form  taxonomical  clusters.  They  also  exhibit  a  mild  tendency  to  form 
functional  clusters.  Taxonomical  clustering  does  not  correlate  with  structural 
clustering,  s\  hile  functional  clustering  correlates  well  with  it. 
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L  INTRODUCTION 


A.  MOTIVATION  FOR  THIS  THESIS 

As  computing  systems  perform  more  and  more  sophisticated  functions, 
the  software  components  of  such  systems  necessarily  become  larger  and 
more  complex.  In  addition,  a  plethora  of  critical  systems,  such  as  nuclear 
power  plants  and  aircraft  control  systems  are  controlled  by  software.  The 
growth  in  size  and  complexity  of  software  results  in  an  even  faster  growth 
in  the  complexity  of  software  testing.  Testing  is  currently  a  very  labor 
intensive  and  expensive  process  that  accounts  for  approximately  50%  of 
software  system  development  (Myers  1979,  Korel  1990). 

The  need  for  reliable  software  operation  is  increasing  rapidly.  This 
implies  that  extensive  software  testing  is  frequently  necessary  despite 
expenses.  This  thesis  deals  with  software  testing,  and  is  intended  as  a  sequel 
to  previous  research  on  failure  region  identification  (Shimeall  etal  1991)  and 
failure  region  clustering  (Ginn,  1991). 
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B.  OUTLINE  OF  THE  PROBLEM 


Failure  regions  appear  to  have  a  tendency  to  cluster  so  the  neighbo¬ 
rhood  of  a  specific  failure  region  may  reveal  more  software  faults  than  the 
one  that  caused  the  region.  An  empirical  study  (Ginn  1991)  on  data  obtained 
by  Shimeall  and  Leveson  (1991)  has  demonstrated  this  clustering  tendency 
of  failure  regions  using  appropriate  clustering  criteria  for  multidimensional 
nominal  types  of  data  (Jain  and  Dubes,  1988).  A  number  of  issues  have  been 
raised  from  diis  research: 


«  Several  failure  regions  demonstrated  a  strong  clustering  tendency  in  one 
dimension  but  weak  clustering  in  other  dimensions.  However  it  is  not 
known  if  this  is  a  coincidence  or  normal  behavior  of  failure  regions. 

•  Software  faults  were  numbered  in  order  as  they  were  discovered,  by  the 
various  testing  techniques  applied  by  Shimeall  and  Leveson  (1991),  so 
that  many  of  the  sequentially  numbered  faults  were  discovered  by  the 
same  detection  technique.  There  was  also  a  strong  tendency  of 
clustering  for  sequential  friults,  but  it  is  as  yet  unknown  if  there  is  a 
correlation  between  certain  detection  techniques  and  certain  types  of 
fault  clusters. 

•  It  is  not  clear  which  types  of  conditions  and  variables  are  more  likely 
to  result  in  clusters. 
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There  is  empirical  evidence  that  known  failure  regions  may  be  used  to 
understand  the  relationship  of  one  fault  to  another.  Failure  regions  offer  a 
mechanism  for  identifying  commcm  features  among  faults,  because  the 
relationship  between  two  failure  regions  corresponds  to  the  relationship  of 
the  code  locations  of  the  associated  faults.  The  goal  of  this  research  will  be 
an  improved  testing  technique  that  incorporates  failure  region  behavior.  To 
do  this,  we  need  to  better  understand  how  parts  of  the  program  interact, 
since  faults  with  similarities  in  their  failure  regions  are  expected  to  occur 
under  similar  conditions. 

C  OVERVIEW  OF  THE  THESIS 

Chapter  11  gives  an  extensive  review  of  preceding  research  on  software 
testing  in  general  and  failure  region  analysis  in  particular.  Chapter  111 
introduces  a  testing  technique  on  clustering  and  presents  empirical  results 
from  the  qrplication  of  the  said  technique  on  the  results  of  a  software 
experiment.  Finally,  Chiq)ter  IV  summarizes  the  conclusions  that  can  be 
drawn  from  the  results  and  offers  suggestions  and  recommendations  for 
further  research. 
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n.  BACKGROUND  AND  RELATED  W(HUC 


This  chapter  reviews  software  testing  definitions  and  methods  for 
dealing  with  software  fiiult  and  failure  association.  Symbols  and  terminology 
are  similar  to  those  used  in  the  classical  piq)ers  (such  as  Weyuker  and 
Ostrand,  1980,  Goodenough  and  Geihart,  1975).  A  brief  overview  of  theories 
of  software  testing  related  to  our  research  is  presented.  Previous  wofk,  botib 
empirical  and  theoretical,  in  die  area  of  failure  regions  analysic  s  included. 
Finally  the  cluster  analysis  technique  that  has  been  used  by  Ginn  (1991)  is 
briefly  presented. 

A.  SOFTWARE  TESTING 
1.  Basie  Definitions 

Testing  is  a  method  of  program  verification  that  deduces  from 
execution  diat  a  program  possesses  required  properties  (Morell  1990). 

The  input  data  for  the  majority  of  programs  come  from  a  multi¬ 
dimensional  space.  An  example  given  by  Amman  and  Knight  (1988)  refers 
to  a  program  processing  an  input  of  20  floating  point  numbers  thus  having 
20  dimensions. 
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We  shall  use  D  for  the  input  (k>main  of  a  program  F,  and  R  for  the 
output  range  of  F.  On  input  d  (d€D)  F,  if  it  terminates,  produces  ou^ut 
F(d)  (eR).  Hence  the  program,  may  be  viewed  as  a  mining  from  the  input 
domain  to  the  output  range. 

The  output  specification  for  F  is  given  by  OUT(i,y),  where  X€D 
and  y€R  F  is  correct  on  input  d  (abbreviated  OK(d))  if  F(d)  exists  and 
OUT(d»F(d)).  A  test  T  (c  D)  is  formally  defined  as  a  subset  of  the  input 
domain  (after  Weyuker  and  Ostrand,  1980,  Goodenough  and  Geriiart,  1975). 
More  often  than  not,  software  fiiilures  appear  at  what  seems  to  be  an  obscure 
set  of  data  or  a  special  case  (Ammann  and  Knight  1988).  This  introduces  an 
additional  complication  to  the  testing  process. 

2.  ideal.  Valid,  and  Reliable  Tests 

An  ideal  test  must  be  valid,  which  implies  that  for  every  fault  in 
program  F  exist  entries  in  test  T  that  cause  the  fault  to  execute  and  produce 
incorrect  output  (Goodenough  and  Geriiart  1975).  Formally: 

IdeaKiy-iVFaubeF:  3d€Tf\^OK{ilf)  (2-1) 

A  rettable  test  is  one  that  either  produces  entirely  correct  output  or 
entirely  incorrect.  Formally: 
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MbMeiT) ~[(^deT. <xm V(V<(e T.  -<OK(m  <2-2) 

The  importance  of  a  failure  region  clustering  theory  becomes  q^arent 
here  from  the  point  of  view  of  test  selection.  It  was  shown  by  Weyuker  and 
Ostrand,  (1980)  that  an  ideal  and  reliable  test  is  exhaustive  therefore  it  is 
desirable,  for  the  sake  of  feasibility,  to  select  a  much  smaller  test  to  the  end 
of  revealing  certain  types  of  finilts.  A  failure  region  clustering  themy 
provides  same  guidelines  for  this  type  of  selection. 

B.  PROGRAM  PATHS 

1.  The  Conecpt  of  a  Pngiam  Path 

A  path  in  a  program  is  defined  as  a  sequence  of  statements  (Rich¬ 
ardson  and  Clarke  1985).  It  is  quite  obvious  that  a  padi  may  be  decomposed 
into  a  number  of  subsidiary  paths.  A  block  statement  is  decomposed  into  its 
constituent  enclosed  statements  whose  execution  depends  upon  evaluation  of 
a  condition  ^ich  is  also  part  of  the  block  statement. 

The  control  flow  statements  in  a  computer  program  partition  the 
input  space  into  a  set  of  mutually  exclusive  domains  each  of  which  causes 
a  corresponding  path  to  be  executed  (White  and  Perera  1986).  This  concept 
of  path  offers  a  natural  way  to  partition  the  input  domain. 
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The  sub<k»nam  Dj  of  path  j,  is  kilned  by  a  boolean  expression 
(say  F(j))  diat  is  the  conjunction  of  the  path's  branch  predicate  constraints. 

In  general  these  predicates  are  expressed  in  terms  of  both  local  (program) 
and  input  variables.  However  it  is  possible  to  replace  each  program  variable 
qjpearing  in  the  predicates  by  its  symbolic  value  defined  in  terms  of  input 
variables  al<mg  that  path  and  get  an  equivalent  constraint  that  is  the  partition 
boundary,  P(j),  as  a  function  of  input  variables  only  (also  predicate 
interpretation  after  White  and  Wiszienwski  1988). 

2.  Programs  as  Sets  of  Paitial  FuncfioBS 

Using  the  concept  of  path,  as  defined  in  the  previous  subsection, 
we  can  model  a  program  as  a  set  of  partial  functions  from  the  input 
partitions  (Dj)  to  the  output  space  (O),  each  of  die  partial  functions 
corresponding  to  the  execution  of  a  sequence  of  statements  along  the 
corresponding  path  (j)  (Richardson  and  Clarke  1985).  Formally: 

Sj.iSi2«S^,(D,)  *  (TcO  (23) 

where  stands  for  the  sequence  of  statements'  along  path  j.  In  the 

case  of  loops  executing  along  the  path  (let  be  the  sub  sequence 

'  The  condition  evaluations  are  included  into  the  sequence  of  statements  for 
consistency  although,  since  the  path  is  pre-determined  they  do  not  affect 
the  result  of  the  operation. 
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of  statements  in  the  loop  body)  the  path  sequence  of  statements  may  be 
written  Sj,,Sp(  This  model  assigns  sequences  that  differ 

only  by  dieir  number  of  loop  iterati<ms  to  the  same  program  path. 

As  a  path  may  be  decomposed  into  other  paths,  an  input  domain 
partitmn  mi^  be  refined  by  the  same  token  in  a  hierarchical  way,  as  new 
branch  predicate  conditions  are  "AND  ed"  to  the  existing.  This  suggests  a 
more  generic  approach  to  the  issue  of  input  domain  partition  than  the  one 
suggested  by  Richardson  and  Clarke  (1985)  where  they  distinguish  this 
partition  into  impiemaitaiioH  and  specification  partitions  and  point  out 
certain  discrepancies  due  to  the  iidierent  differences  of  the  specification 
versus  the  implementation  languages. 

3.  Regular  Expres^ns  for  Paths 

The  set  of  paths,  on  a  flow-chart,  can  be  expressed  in  algebraic 
form  (Beizer  1990).  Path  expressions  are  converted  to  regular  expressions 
that  can  be  used  to  examine  structural  properties  of  program  paths. 

For  any  single  path,  j,  of  the  program,  the  sequence  of  statements 
introduced  in  the  previous  section  is  the  path  product  of  j.  Path 
products  are  also  defuied  on  path  segments.  The  path  that  consists  of 
successive  path  segments,  has  a  path  product  equal  to  the  concatenation  of 
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their  path  products.  A  set  of  parallel  paths  between  two  nodes  has  a  path 
product  equal  to  the  sum  of  the  path  products  of  the  parallel  paths. 

Condition  evaluations  are  retained  in  the  path  products  for 
consistency  (cf  previous  subsection).  In  fact,  the  path  products  of  the  two 
segments  starting  fiom  a  decision  point  (such  as  an  if  C  then  else  construct 
or  a  loop  exit  condition  etc)  are  preceded  with  the  condition  C  and  condition 
-iC  respectively,  depending  upon  \i1)ich  part  of  the  decision  (if  or  else) 
results  in  their  execution.  The  regular  expressions  for  paths  provide  a  concise 
and  compact  notation  for  both  the  path  conditiom  (which,  for  a  given  set  of 
paths,  are  derived  by  conjunction  of  all  conditions  on  successive  path 
segments  and  disjunction  of  conditions  on  parallel  path  segments)  and  path 
acdom  (^^ch,  in  the  previous  secticm,  are  modelled  as  partial  functions 
from  the  input  domain  to  the  output  range). 

C  ON  ERRORS  AND  FAULTS 

A  fiudt  is  an  erroneous  piece  of  program  source  code,  while  an  error 
is  a  discrepancy  between  a  computed  value  and  the  true,  specified,  or 
theoretically  correct  value. 
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A  fiMure  specifies  the  inability  of  a  module  to  perfonn  its  specified 
function  and  includes  both  erroneous  output  and  failure  to  produce  output 
(see  ANSI-IEEE  STD  610.12-1990). 

A  fault  (E)  is  formally  modeled  as  a  3-tuple  (Shimeall  et  al  1991): 

E  =  (2.4) 

^^iiere  L,  is  the  location  of  the  Oiult^  (^iiich  is  some  program  statement),  V, 
the  list  of  variables  that  form  the  error  caused  by  the  fault  and  C,  is  a 
Boolean  condition  under  which  the  fault  is  activated.  An  interesting  point  is 
that  a  feult  activation  does  not  always  imply  a  failure  (coincidental 
correctness.  Morel  1988).  There  are  throe  conditions  that  must  hold  true  for 
a  fault  to  produce  a  failure  (Shimeall  et  al  1991)  which,  apart  from  Cp  are 
the  reachabiUty  condition  and  the  error  propitiation  condition. 

A  basic  assumption  about  the  faults  in  code  is  the  competent  program¬ 
mer  hypothesis  (DeMillo  et  al  1978).  This  states  that  a  competent  program¬ 
mer  will  write  a  program  that  is  syntactically  close  to  the  correct  program. 

A  taxonomy  of  the  types  of  faults  possibly  found  in  a  program  is  given 
by  Beiser  (1990): 

^  Sometimes  a  fault  is  "distributed”  to  more  than  one  locations  while 

certain  types  of  faults,  such  as  missing  functions  etc.,  do  not  have  a  well 
defined  location. 
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].  RcqiiRiiiciits  and  Spacificadons 

These  include  incomplete  or  self-contradictory  specifications, 
missing,  wrong  or  superfluous  features,  and  not-well-specified  feature 
interactions. 

2.  Stnictuial  Emus 

Such  as  control  and  sequence  errors,  logic  errors,  incorrect 
fmrmulae  applications,  use  of  uninitialized  variables.  These  can  be  further 
divided  (Richardson  and  Clarice  1985)  into: 

A  ComputadoH  Enois 

A  computation  error  occurs  when  the  correct  path  through  the 
program  is  taken,  but  the  ouq>ut  is  incorrect  because  of  faults  in  the 
computation  along  the  path. 

b.  Domain  Errors 

A  domain  error  (White  and  Perera  1986)  occurs  ^en  a  specific 
input  follows  the  wrong  path  due  to  an  error  in  the  control  flow  of  the 
program.  They  are  of  two  kinds: 
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(1)  Missing  Path  Errors:  They  occur  when  a  special  case 
requires  a  unique  sequence  of  actions,  but  the  program  does  not  contain  a 
corresponding  path. 

(2)  Path  SetecdoH  Errors:  They  occur  when  the  program 
recognizes  the  need  for  a  path,  but  incorrectly  determines  the  conditions 
under  which  the  path  is  executed. 

3.  Data  Enois 

Occur  uben  a  specific  input  follows  the  correct  path,  but  an  error 
such  as  wrong  data  declarations,  wrong  data  initialization  (especially  in 
shared  dynamic  objects),  etc.  results  in  erroneous  output. 

4.  Coding  Emus 

The  most  common  coding  errors  are  documentation  inconsistencies, 
typographical  errors,  and  erroneous  use  of  a  program  statement  when  its  side 
effects  are  not  well  understood. 

5.  Iirteiface  Eirois 

Such  as  interface  communication  problems,  incorrect  input-ou^ut 
format,  wrong  subroutine  control  sequence,  wrong  call  parameters,  incon¬ 
sistent  entry  or  exit  parameter  values,  etc. 
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D.  ON  FAILURE  REGIONS 


A  software  Jblbire  reg^n  (GcD)  is  the  set  of  all  input  values  that  are 
mapped  by  an  individual  program  fault  onto  any  failure  (or  onto  a  failure  set, 
as  in  figure  1.1).  It  is  noted  that  die  concept  of  failure  region  includes  both 
the  input  points  which  cause  erroneous  output  and  the  geometry  of  it.  These 
sets  are  always  finite,  since  the  number  of  representations  in  a  finite  machine 
is  limited,  but  more  often  than  not  intractably  large.  The  failure  region  boun¬ 
daries  are  defined  by  Boolean  conditions  on  the  input  domain. 

Shimeall  et  al  (1991)  demonstrate  a  technique  that  analytically  de¬ 
termines  the  three  conditions  for  a  known  fault  (which  include  all  conditions 
for  reaching,  activating  and  propagating  the  fault),  by  synAoUcaity  executing 
(King  1976)  the  source  code  on  every  "loop  [0,1]"  path  (Loops  are  handled 
as  in  previous  section,  by  {qiplying  either  the  exit  condition  or  the  loop 
effects)  and  conjuncting  the  obtained  Boolean  expressions.  The  conjunction 
of  those  conditions  is  the  mathematical  specification  of  the  failure  region 
boundary  (which  may  be  called  Bouiid(G))  subject  to  the  limitations  of  finite 
representation  in  computing  machines. 
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Figure  1.1 


AfiSGdationB  between  Failiire  regions  Faults 
and  Failures  (adapted  from  Shimeall  et  al 
19^1) 


E.  BOUNDS  CORRELATION  BETWEEN  DIFFERENT  FAILURE 
REGIONS 

The  dimensions  of  the  input  domain  of  a  program  can  provide  a  set  of 
criteria  for  the  correlation  between  different  failure  regions.  To  this  end  a 
classification  of  these  dimensions  is  used  (Ginn  1991),  with  respect  to  any 
pair  of  failure  regions: 
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•  The  dimensions  that  appear  in  both  boundaries  in  exactly  the  same  way 
are  teimed  idendcalty  parddpatlng  dimensions. 

•  Those  who  appear  in  both  boundaries  but  not  in  an  identical  way  are 
tenned  coinddentaiiy  participating  dimensions. 

•  The  dimensions  that  do  not  tqjpear  in  the  boundaries  of  both  regions  are 
called  nonbounding  dimensions,  because  the  boundaries  these  dim¬ 
ensions  place  on  failure  regions  are  no  more  restrictive  than  their  entire 
range  of  values. 


In  Ginn  (1991)  the  Identical  and  Coincidental  dimensions  are  collective¬ 
ly  referred  to  as  Composite  dimensions. 
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DL  CLUSTER  ANALYSIS  AS  A  PREDICTOR  FOR  FAULTS 

A.  CLUSTER  ANALYSIS 

The  identification  of  groups  that  have  similar  characteristics  is  the  goal 
of  clustering  analysis.  In  the  context  of  this  thesis,  the  objective  of  failure 
region  analysis  is  to  investigate  clustering  of  failure  regions  and  identify  the 
logical  relation  of  the  program  locations  of  the  faults  responsible  for  these 
failure  regions. 

By  definition  of  failure  region  (Shimeall  et  al  1991  for  example)  the 
execution  of  a  program,  with  data  from  some  failure  region  as  input,  will 
reackj  activate^  and  propagate  to  the  output  the  corresponding  program  fault. 
The  conjunction  of  predicates  that  defines  the  bound  of  the  region  (Chapter 
II-D)  contains  a  statement  of  the  reachability,  activation  and  propagation 
conditions  for  the  corresponding  fault. 

The  criterion  used  by  Ginn  (1991)  assumed  that  clustering  between 
failure  regions  occius  >^en  identical  or  similar  predicates  appear  in  the 
bounds  of  these  failure  regions.  Therefore,  one  could  use  identical  dim¬ 
ensions  (corresponding  to  identical  predicates)  or  composite  dimensions 
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(corresponding  to  both  identical  and  similar  predicates)  to  define  a  measure 
of  clustering.  The  assumption  made  by  Ginn  (1991)  is  reasonable,  because 
it  is  expected  that  when  failure  regions  correspond  to  faults  that  share  some 
program  padis  (therefore  some  of  the  predicates  in  common)  it  is  possible, 
but  not  necessary,  that  the  same  input  may  reveal  all  of  them.  On  the 
contrary,  >^en  the  faults  do  not  share  any  path,  it  is  impossible  for  both  of 
them  to  be  revealed  by  the  same  input. 

The  criteria  used  for  the  failure  region  ^  unds  variables  are  essentially 
ordinal:  identical,  coincidental,  nonbounding,  in  descending  order.  Therefore 
relative  values  cannot  be  assigned  to  them.  There  are,  however,  two  different 
coefficients  that  may  serve  as  a  measure  of  clustering  for  ordinal  data  (see 
Jain  and  Dubes  1988): 

S(G,Gj) (3.1) 

The  simple  matcUi^  coefficients  for  two  failure  regions  G„  Gj  are 
defined  by  3.1,  where  stands  for  the  non-bounding  dimensions  of  both 
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failure  regions  a,i  for  the  shared^  dimensions  of  Gi  and  Gj,  (cf  Chapter  II>E) 
and  a  for  the  total  number  of  input  dimensions.  The  similarity  of  failure 
regions  G|  and  is  greater  the  closer  the  simple  matching  coefficient  is  to 
unity,  since  S(G|,G|>»1,  because  The  inclusion  of  in  the 

numerator  of  the  coefficient  emphasizes  equally  bounding  and  non-bounding 
dimensions  and  can  result  in  very  high  (close  to  unity)  values  of  the 
coefficient  \\iien  the  total  number  of  input  dimensions  is  much  greater  than 
die  bounding  dimensions  of  two  failure  regions.  Suppose  for  example  the 
pair  of  failure  regions: 

BaundiGi)  =(ar<5)A(y<8)A(^<10)  02) 

Bound(G^  =  (w<0)A(i<<0)  03) 

If  the  total  number  of  dimensions  is  250  then  i^q«245,  ai,Ml,  and 
S(G„(^>»245/25(M1.98,  \^ich  implies  a  high  degree  of  clustering  while  it 
is  obvious  that  the  failure  regions  are  not  related. 


The  term  shared  may  refer  to  dimensions  appearing  in  identical  predicates  in  two 
failure  regions  definitions,  or  may  include  both  similar  and  identical  predicates.  Ginn 
(1991)  investigates  both  cases  separately. 
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The  Jaccord  coefficients,  defined  by  3.4,  seem  more  sensitive  to 
clustering  since  tbey  emphasize  more  the  composite  dimensions,  as  opposed 
to  die  simple  matching  coefficients  that  are  symmetric  in  composite  an  non- 
coincidental  dimensions.  As  with  die  simple  matching  coefficients,  the 
similarity  of  failure  regions  and  Gj  is  greater  the  closer  the  Jacard 
coefficient  is  to  unity,  since  J(G|,G|>»1. 

AGpG)  =  (34) 

^  "-‘Ho 


However,  under  certain  circumstances,  the  reliability  of  Jaccard 
coefficients  is  debatable.  Suppose  for  example  the  pair  of  failure  regions 
defined  by  (3.5),  (3.6): 

ltoii#K/(Gi)=(;c<5)A(y<8)A(z<10)  (3^) 

Bound(G2)  =  (x<5)A(y<8)A(z<10)A(w<0)A(ii<0)  (3.6) 
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In  this  case  die  Jacard  coefficient  J(G„G2)*3/(l+2'fO>^.6,  nliile  it  is 
intuitively  obvious  that  the  relation  is  probably  stronger  than  is  suggested  by 
the  coefficient.  Chi  the  other  hand,  the  simple  matching  coefficient, 
depending  upon  the  number  of  dimensions  that  are  non-bounding  for  both  Gj 
and  G,  may  vary  between  0.6  and  0.9999. 

In  the  following  sections,  we  shall  examine  how  pairs  of  faults  may 
result  in  clustered  failure  regions,  as  well  as  how  the  Jaccard  coefficients  are 
affected. 

B.  RELATED  FAULTS 

The  goal  in  analyzing  fault  relations  is  to  understand  the  clustered 
failure  regions,  which  appear  in  software  experiments  (Ginn  1991).  To  this 
end,  it  is  required  to  determine  i^t  kind  of  ftiult  relationship  results  in 
clustered  failure  regions. 

1.  Logical  Rdalion  of  Faults 

Faults  can  be  logically  r^aed  if  they  are  either  the  same  logical 
flaw  (TaxonomicaUy  rdated)  or  they  are  located  in  regions  of  programs  diat 
compute  the  same  part  of  the  application  {Funclionalfy  Rdaied). 
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2.  TaxoBOHiicayy  RdalMi  Fairiti 


A  logical  flaw  is  an  error,  or  misunderstanding  by  the  programmer, 
in  the  design  logic  of  a  program  that  results  in  a  number  of  faults.  Taxo> 
nomically  related  fliults  are  expected  to  be  both  application  dependent  and 
programmer  dependent.  They  are  iq>plication  dependent  because  it  is 
e^qiected  fliat  iq^lications  requiring  a  great  number  of  a  certain  type  of 
constructs  (say  loops  or  if  statements  for  example)  are  more  prone  to  die 
type  of  faults  peculiar  to  these  constructs  (loop  iteration  and  control  flow 
faults  respectively  for  fliis  example).  It  is  not  always  clear  fiom  the 
specification  of  a  program  exactly  how  many  such  constructs  will  be 
required,  and  it  is  usually  up  to  the  programmer  to  decide  for  die  im¬ 
plementation  details.  However,  the  higher  level  design  of  a  program  always 
gives  an  indication  whether  the  implementation  requires  many  iterations  or 
a  lot  of  case  handling  etc.  Therefore  this  type  of  faults  depends  both  on  the 
design  and  the  implementation  details. 

On  die  other  hand  they  are  programmer  dependent  because  every 
programmer  has  his/her  own  weak  and  strong  points  in  developing  a 
software  design,  and  it  is  obvious  that  some  phases  of  his/her  work  will  be 
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more  prcme  to  fiuilts  Uian  olliers.  This  also  holds  true  for  both  the  design 
and  the  implementaticm  phases. 

3.  TaKOBOBiical  Faidt  ClanHkiriioB  of  •  Sofitwaic  Experiment 

The  classificatiiMi  of  die  known  (nogram  faults  in  a  set  of  eight 
redundant  versicms  of  a  combat  simulation  program  (Shimeall  1991a,  1991b) 
constructed  as  part  of  a  software  experiment  (Shimeall  and  Leveson  1991) 
were  analyzed.  The  fault  categories  used  in  this  taxonomy  are  from  Beiser 
(1990).  According  to  his  classification  scheme  each  fault  is  characterized  by 
the  type  of  logical  flaw  that  produced  it  (for  example  case  selection  bug, 
control  logic  bug  etc)  and  is  assigned  a  four  digit  code  number. 

The  first  digit  is  characteristic  of  the  highest  level  of  the  taxonomy 
hierarchy  (i.e..  Structural  Bug,  has  code  3xxx)  while  the  last  digit  specifies 
the  exact  category  of  the  fault  (for  example  a  Structural,  control  state  fault, 
has  code  3154).  The  advantages  of  this  taxonomy  is  that  provides  an  easy 
hierarchical  and  logical  scheme,  and  the  four  digits  of  the  code  specify  the 
four  levels  of  the  used  hierarchy  (cf.  Chapter  II  section  C  for  a  discussion 
of  the  highest  levels). 

The  results  of  die  friult  classification  in  the  eight  versions  are 
presented  in  Table  3.1  (detailed  results  including  fault  relative  frequency 
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histograms  for  the  eight  versions,  and  for  the  observed  to  expected  relative 
frequency  ratio  of  the  total  number  of  faults  are  presented  in  Appendix  B, 
Figures  B.l  to  B.8).  In  addition  to  the  &ult  statistics,  in  the  table  we  include 
the  number  of  statements,  if  and  case  selection  statements,  and  loop 
constructs. 

4.  The  'Taxonomkal  Clusleiiiig**  Hypothesb 

The  testing  of  the  '\axonoimcal  clustering''  hypothesis  is 
performed  by  comparison  of  the  actual  results  with  a  simple  random  model 
that  assumes  that,  given  the  total  number  of  faults  in  a  program,  the 
probability  of  an  fault  occurrence  in  any  line  is  equal  to  the  ratio  of  the  total 
number  of  faults  by  the  total  number  of  lines.  The,  ratio,  is  the  expected  fault 
rate  for  the  given  program.  The  expected  number  of  faults  for  each  of  the 
categories  in  Table  3.1,  for  the  random  fault  distribution,  is  calculated  as  the 
product  of  the  fault  rate  time  the  number  of  statements  where  this  type  of 
fault  can  occur. 

Within  the  scope  of  our  working  model,  processing,  initialization 
and  algorithmic  faults  (codes  32 lx,  322x,  323x)  occur  in  all  program 
statements  apart  from  if,  case  and  loop  constructs.  Loop  and  iteration  faults 
(code  314x)  occur  in  loop  constructs.  Control  logic,  case  selection  and 
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control  state  faults  (code  312x,  313x,  315x)  may  appear  in  if  or  case 
statements.  The  exception  handling  faults  tqjpear  usually  either  in  control  (if, 
case)  statements  or  loop  exit  conditions.  Therefore  we  assume  further  that, 
on  the  average,  half  of  the  faults  in  loop  and  control  statements  are  on 
exception  handling  and  the  other  half  on  loop,  iteration  and  control 
re  S5?ectively. 

The  random  fault  allocation  that  results  from  the  use  of  our  model, 
is  also  included  in  Table  3.1  for  th^  eight  program  versions.  In  this  case  the 
’’random"  version  number  has  the  sai>script  r. 

In  Appendix  B  we  present  the  relative  frequency  distribution  for 
both  the  actual  number  of  faults  and  the  model  predictions.  To  test  the 
goodness  of  fit  of  the  observations  to  the  random  fault  distribution  model  we 
make  use  of  the  Chi-Square  (x^)  OneSanqfle-Test  (Siegel  and  Castellan 
1988).  This  test  gives  the  level  of  signiftcance  (probability  of  occurrence  in 
frict)  that  the  statistic,  wfrich  increases  with  the  difference  between  the 
observed  and  the  expected  fault  distribution,  is  greater  than  a  certain  value. 
The  higher  the  probability  of  the  said  difference  the  more  confident  we  are 
that  the  selected  model  corresponds  to  the  actual  distribution.  However,  the 
results  of  Table  3.1  imply  that  the  probability  of  the  observed  differences 
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between  the  actual  number  of  faults  and  the  model  prediction  is  below 
0.001,  because  for  3  degrees  of  freedom  (since  there  are  four  fault  cate¬ 
gories)  the  smallest  value  of  was  19.7  in  version  VI.  Therefore  we  can 
conclude  that  the  fault  distribution  is  not  uniform.  This  implies  that  the  faults 
that  have  the  most  opportunity  of  being  committed  are  not  always  the  most 
fiequent  in  a  program. 

It  is  evident  from  the  statistics  in  Table  3.1  that  control  logic,  case 
selection  and  control  state  faults  tqtpear  at  a  rate  about  fourteen  times  the 
average  friult  ratio  in  the  program  w^ile  processing  and  initialization  fault 
rate  is  about  30%  of  the  average.  This  leads  to  the  conclusion  that  the 
handling  of  the  control  logic  of  the  program,  at  least  in  the  CONFLICT 
experiment  (Shimeall  1991a,  1991b,  Shimeall  and  Leveson  1991)  and  for  die 
programmers  selected,  represents  a  task  of  much  greater  difficulty  than 
processing  and  initialization  in  the  same  program. 
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TABLE  3.1 

(PROGRAM  STATBTICS,  FOR  TAXWOMICAL  CLUSTERING) 
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TABLE  3.1 

(PROGRAM  STATISTICS,  FOR  TAXONOMiCAL  CLUSTERING) 
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11999 

7.51 

10.8 

209.4 

18.2 

239 

X^ 

- 

- 

- 

0.8 

1741.0 

105.4 

2.6 

1843 
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5.  TflxoBonkal  Onsteriog  of  Fiiliiic  Rcgioiis  Complied  to  Ifae 

Stractnnl  Cloiteriiig  of  Failiiie  Regions 

In  this  section,  we  examine  whether  the  logical  clustering  of  failure 
regions  correlates  with  the  number  of  shared  dimensions  in  their  bounds.  The 
number  of  shared  dimensions  at  the  bounds  of  two  failure  regions  has  been 
used  in  the  definition  of  a  clustering  metric,  the  Jaccard  coefficient  between 
two  regions,  by  Ginn  (1991).  On  the  other  hand,  we  consider  two  failure 
regions  as  members  of  the  same  Taxonomical  cluster  if  the  associated  faults 
are  both  results  of  similar  logical  flaws.  We  classify  the  Taxonomical 
clusters  as  follows: 

•  "type  A.  Corresponds  to  loop  and  iteration  faults  (code  314x). 

•  "type  B.  Corresponds  to  control  logic,  case  selection  and  control  state 
(codes  312x,  313x,  315x)  faults. 

•  Type  C:  Corresponds  to  processing,  initialization  and  algorithmic  faults 
(codes  32 lx,  322x,  323x). 

•  Type  D:  Corresponds  to  faults  in  exception  handling  (code  316x). 

Summary  statistics  of  this  comparison  are  presented  in  Table  3.2 
(Details  are  included  in  Appendix  D).  From  these  results  it  is  apparent  that 
the  appearance  of  shared  dimensions,  and  the  variation  of  the  Jaccard 
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i 
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coefficient,  on  failure  region  bounds  does  not  depend  on  the  ta^onomical 
clustering  as  analyzed  in  subsection  b,  where  the  clustering  criterion  between 
different  failure  regions  has  been  whether  they  correspond  to  faults  resulting 
from  the  same  logical  flaw. 


TABLE  32 

AVERAGE  AND  STANDARD  DEVIATION  OF  THE  JACCARD 
COEFnCIENT  BETWEEN  TAXONOMICALLY  CORRELATED 

FAILURE  REGIONS 


VER¬ 

SION 

■KlMilMlr 
oTtjr^  A 

Fsiittic  region 
pair  of  type  B 

Failure  region 
pair  of  type  C 

Failure  region 
pair  of  type  D 

UBMiKiiaee 

I 

- 

0.05 1±0. 115 

0.04910.134 

0.25^ 

0.04210.108 

n 

- 

0.050Hh0.149 

0.133 

- 

0.02910.066 

IHQH 

- 

0.076±0.135 

0.05410.107 

0 

0.05510.115 

IV 

- 

0.032±0.061 

0.05010.118 

0.077 

0.03610.092 

V 

- 

0.01910.085 

0.01710.034 

- 

0.00810.035 

VI 

- 

0.07310.230 

0.05910.118 

- 

0.01810.039 

vn 

- 

0.07410.174 

0.03110.075 

0.11110.19 

0.01810.065 

vm 

- 

0.12310.429 

- 

0.13410.181 

0.11710.227 

In  Table  3.3  we  present  the  same  statistics  as  in  Table  3.2  with 
the  difference  that,  in  this  case,  we  use  the  average  non-zero  Jaccard 
coefficient  between  taxonomically  correlate*’  failure  region.  This  indicates 


In  entries  without  Standard  Deviation  there  is  only  one  value. 
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whefter  the  taxonomical  clustering  of  failure  regions  correlates  with  the 
number  of  shared  dimensions  in  their  bounds,  when  these  shared  dimensions 
exist. 


TABLE  33 

AVERAGE  AND  STANDARD  DEVIATION  OF  THE  NON  ZERO 
JACCARD  COEFFICIENT  BETWEEN  TAXONOMICALLY 
CORRELATED  FAILURE  REGIONS 


A 

Failuie  re^on 
pair  of  type  B 

F^luie  region 
pair  of  type  C 

Failure  region 
pair  of  type  D 

Ufimilalia 
fUtaR  MflM  piir 

I 

- 

0.124±0.150 

0.24710.361 

0.250 

0.16410.159 

n 

- 

0.39310.196 

0.133 

- 

iHipiM 

- 

0.17110.139 

0.13910.134 

- 

0.15910.143 

IV 

- 

0.12910.048 

0.15610.167 

0.077 

0.13110.136 

V 

- 

0.31510.172 

0.067 

- 

0.11510.089 

VI 

- 

0.73 

0.19110.147 

- 

0.08610.026 

vn 

- 

0.43410.147 

0.13310.109 

0.333 

0.17310.110 

vm 

- 

0.46910.222 

- 

0.28310.117 

0.38710.258 

The  results  in  Table  3.3  provide  us  with  evidence  in  favor  of  the 
intuitively  obvious  hypothesis  that  the  structural  clustering  of  failure  regions 
does  not  depend  on  the  taxonomical  clustering.  This  result  can  be  attributed 
to  the  fact  that  structural  clustering  of  failure  regions,  depends  strongly  on 
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the  control  and  data  flow  structure  of  the  program,  with  respect  to  the 
corresponding  fault  location.  Chi  the  other  hand  the  taxonomical  clustering 
dependence  is  restricted  to  the  type  of  statement  at  the  fault  location. 

6.  The  Case  of  Ftmclionally  Related  Faults 

The  hypothesis,  that  logically-related  faults  located  in  regions  of 
programs  that  compute  the  same  part  of  the  ^plication  may  lead  to  some 
type  of  clustering,  is  based  on  the  intuitively  obvious  assumption  that  some 
parts  of  a  problem  may  be  more  difficult  to  handle  or  more  "error  prone" 
than  others  (Brilliant  et  al  1990).  We  shall  call  this  type  of  faults  Junction 
naify  rdaUd. 

7.  Functional  Fault  Qassification  of  a  Software  Experiment 

The  distribution  of  the  known  program  faults  in  the  eight  versions 
of  the  combat  simulation  program  (Shimeall  1991,  1991b)  to  the  program 
modules  implementing  different  functional  requirements  of  the  specification 
was  analyzed.  We  classified  the  functional  clusters,  according  to  the 
CONFLICT  Specification  (Shimeall  1991b)  as  follows: 

•  jype  /:  Positioning  and  Movement 
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•  "type  U:  Observation 

•  "type  Hi:  Attrition 

•  IV\  CommunicatiiMi 

•  Type  V:  Environment 

•  "type  U:  Restoration 

•  T)fpe  O'.  Others  (Includes  the  main  procedure  of  CONFLICT  and  some 
procedures  for  initialization  and  output  format) 

8.  The  Functioiial  Clustering  Hypothesis 

The  null  hypodiesis  diat  fault  distribution  is  uniform  over  the 
program  locations  was  used  in  this  analysis  to  test  the  clustering  hypothesis. 
The  number  of  expected  faults  for  each  functional  requirement  of  the 
specification  was  set  equal  to  die  total  number  of  lines  of  routines  im< 
plementing  the  requirement  times  the  fault  rate  (as  in  subsection  lb)  for  the 
program. 

The  results  of  this  analysis  are  presented  in  Table  3.4.,  while 
detailed  analysis  is  included  in  Appendix  D. 

To  test  the  goodness  of  fit  of  the  observations  to  the  random  fault 
distribution  model  we  make  use  of  the  Chi-Square  (x^)  One-Sanqfle-Test 
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(Siegel  and  Castellan  1988),  as  in  subsection  4.  In  this  case  there  are  seven 
categories  of  faults,  therefore  the  degrees  of  freedom  are  six. 

Apart  fiom  versions  IV,  VII  and  VIII,  the  confidence  level  for  the 
null  hypothesis  is  varying  from  1%  to  31%.  In  versions  IV,  VII  and  VIII  this 
confidence  drops  below  0.1%.  This  result,  however,  may  be  attributed  to  the 
low  (less  than  five)  expected  number  of  type  I,  II,  III,  V,  VI  and  O  faults  for 
version  IV  and  the  low  e}q>ected  number  of  type  I,  V,  VI  faults  for  version 
VIII.  In  version  VII  the  result  may  be  attributed  to  the  low,  3.1,  number  of 
e7q)ected  fault  at  column  IV  compared  to  the  10.5  observed. 

The  above  discussion  can  be  verified  in  case  we  consider  only  two 
fault  categories  for  version  IV,  Type  IV  with  7  observed  and  6.7  expected 
faults,  and  all  others  with  16  observed  and  16.3  expected  faults.  This  will 
result  in  a  equal  to  0.019  and  a  corresponding  confidence  level,  one 
degree  of  freedom  this  time,  greater  than  90%. 

In  the  case  of  version  VIII,  we  may  consider  the  fault  categories 
II,  III,  IV,  O  and  all  others,  with  four  degrees  of  freedom,  and  the  x^  will  be 
equal  to  14.85,  which  gives  a  confidence  level  of  1%. 

The  confidence  level  increases  accordingly  with  all  the  remaining 
versions  if  we  group  together  all  categories  with  expectation  value  less  than 
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5  as  suggested  by  Siegel  and  Castellan  (1988).  Under  the  circumstances,  we 
cannot  either  accept  or  reject  the  null  hypothesis  of  uniform  distribution  of 
faults. 

An  alternate  hypodiesis,  that  die  faults  are  uniformly  distributed 
between  type  IV  (Communication),  type  III+II  (Attrition  and  Observation) 
and  all  other  types  is  tested  in  Table  3.5.  The  alternate  hypothesis  assumes 
that  some  type  of  functional  clustering  exists,  because  2/3  of  the  faults  are 
clustered  in  the  Communication,  Attrition  and  Observation  modules,  which, 
on  the  average,  constitute  the  50%  of  die  total  lines  of  code. 


TABLE  3.4 

(PROGRAM  STATISTICS,  fOR  FUNCTIONAL  CLUSTERING) 
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TABLE  3.4  (PROGRAM  STATISTICS,  FOR  FUNCTIONAL  CLUSTER¬ 
ING) 
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TABLE  3^  (ALTERNATE  HYPOTHESIS  TEST) 


TABLE  3^  (ALTERNATE  HYPOniESIS  TEST) 


VtnioB 

Type  IV 

Type 

<HI+V+VI 

TOTAL 

CoafldcMC  Level 

vm 

15 

14 

12 

41 

- 

vm. 

13.67 

13.67 

13.67 

41 

- 

X* 

0.13 

0.01 

0.20 

843% 

TOTAL 

86 

78 

83 

247 

- 

TOTAU 

82.33 

82.33 

82.33 

247 

- 

X* 

0.16 

0.23 

0.01 

mmi 

82% 

The  confidence  levels  of  die  chi-square  test  range  from  84.3%  to 
0.6%.  Although  we  cannot  accept  or  reject  the  alternate  hypothesis,  we 
notice  that  the  confidence  levels  for  acceptance  are  in  general  higher  that 
those  of  the  null  hypothesis.  The  results  for  versions  III,  VI,  VII  and  VIII  are 
in  hivor  of  die  alternate  hypothesis,  which  implies  some  type  of  functional 
clustering.  The  results  for  versions  II,  IV,  V  are  not  in  favor  of  the  alternate 
hypothesis  because  the  functional  clustering  tendency  is  stronger  dian  the 
assumed  by  the  alternate  hypothesis,  and  only  version  I  implies  diat  there  is 
not  any  clustering  tendency  favoring  Attrition  Communication  and  Observa¬ 
tion. 


f 
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Therefore  it  is  reasonable  to  conclude  fiom  the  data  at  hand  diat  there  is 
indeed  some,  not  very  strong,  tendency  of  the  failure  regions  to  cluster  in 
some  groups  of  functional  program  modules  more  than  in  others. 

9.  OoBteriiig  of  FaUmc  Regions  of  Foncfionally  Related  FanHs  Com- 
pnifd  to  the  StnicInnI  Clnstering  of  Failure  Regions 

In  this  section,  we  examine  whether  the  functional  clustering  of 
ftilure  regions  correlates  with  the  number  of  shared  dimensions  in  their 
bounds.  In  this  case  we  consider  two  failure  regions  as  members  of  the 
same  functional  cluster  if  the  associated  faults  appear  in  regions  of  the 
program  that  compute  the  same  part  of  the  application.  Summary  statistics 
of  this  comparison  are  presented  in  Table  3.6  and  3.7  (the  latter  corresponds 
to  the  non  zero  Jaccard  coefficient  case)  while  details  are  included  in 
Appendix  D. 

From  the  results  of  Table  3.6  it  is  not  possible  to  conclude  that  die 
appearance  of  shared  dimensions,  and  the  variation  of  the  Jaccard  coefficient, 
on  failure  region  bounds  does  or  does  not  depend  on  the  fimctional 
clustering,  as  defined  in  subsection  8,  because  the  calculated  standard 
deviations  exceed  die  averages  due  to  the  abundance  of  zero  data  values. 
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However,  from  Table  3.7,  we  can  see  that  the  average  non  zero 
Jaccard  coefficient  is  systematically  higher  for  functionally  correlated  pairs 
that  for  uncorrelated.  This  implies  that,  given  the  structural  clustering  of  two 
failure  regions,  the  clustering  metric  is  higher  for  functionally  correlated 
ones.  This  result  can  be  attributed  to  the  fact  that  structural  clustering  of 
failure  regions  depends  on  the  control  and  data  flow  structure  of  the 
program,  with  respect  to  the  corresponding  fault  location. 

It  is  expected  that  in  a  reasonably  well  structured  program,  faults 
on  the  same  group  of  functional  modules  will  share  a  common  control  and 
data  flow  path  segment  more  often  than  faults  on  different  groups. 
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TABLE  3.6 

AVERAGE  AND  STANDARD  DEVIATION  OF  THE  JACCARD 
COEFFICIENT  BETWEEN  FLNCIKK^ALLY  CORRELATED 

FAILURE  REGIONS^ 


vmioN 

Pi«. 

Ml. 

MM. 

r.i. 

Fin. 

Mr«r 

Mr«r 

1>p«l 

Mr«r 

1>p.IV 

MraT 

TTWV 

Mr«r 

Tmvt 

Mr«r 

l>ptO 

hM 

1 

- 

0.025 

±0.04 

0.140 

±0.006 

0.042 

0.240 

±0.49 

witm 

0.099 

±0.19 

0.0284 

±0.081 

n 

0.091 

±0.24 

- 

0.076 

±0.184 

0.111 

±0.192 

- 

- 

0.0181 

±0.073 

m 

0.2S0 

0.223 

±0.312 

0.097 

±0.137 

0.148 

±0.091 

0 

0.0% 

±0.14 

0.046 

±0.095 

IV 

- 

0.0374 

±0.064 

0.056 

±0.096 

0.0823 

±0.235 

- 

- 

- 

0.019 

±0.053 

V 

- 

0±0 

0.714 

0.0227 

±0.086 

0 

0 

B 

0.0087 

±0.051 

VI 

- 

0.336 

±0.043 

0.1142 

±0.220 

0.075 

±0.127 

- 

« 

0.0643 

±0.16 

0.0184 

±0.071 

vn 

- 

0.0263 

±0.058 

0.2922 

±0.239 

0.131 

±0.220 

0.333 

0 

0.117 

±0.19 

0.0192 

±0.071 

vm 

0.333 

±0.315 

0.75 

0.102 

±0.206 

0.187 

±0.285 

- 

- 

0.069 

±0.180 

In  entries  without  Standard  Deviation  there  is  only  one  value. 
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TABLE  3.7 

AVERAGE  AND  STANDARD  DEVUTIQN  OF  THE  NON  ZERO 
JACCARD  COEFnClENT  BETWEEN  FUNCTIONALLY 
CORRELATED  FAILURE  REGIONS' 


vnsioN 

HI. 

rutaR 

rai. 

rmrnt 

F*w 

FiihK 

Fai. 

Hr«r 

Hr«f 

T>p.O 

MrgT 

Hr«r 

T>»*IV 

Plifrar 

Mr  of 
1>F*V1 

PMrar 

Mr 

Umot- 

1 

- 

0.074 

11 
o  d 

0.042 

A 

Q 

0  350± 

Q  212 

0.430 

1 

0.110 

0.121 

10.131 

n 

- 

0.636 

- 

0.4071 

0.216 

0.333 

- 

- 

0.2061 

0.152 

m 

0,250 

- 

0.5861 

0.189 

0.2441 

0.168 

0.1481 

0.091 

- 

0.240 

1 

0.149 

0.1451 

0.116 

IV 

- 

0.127± 

0.053 

0.167 

0.70 

- 

- 

- 

0.1031 

0.080 

V 

• 

- 

- 

0.2981 

0.160 

- 

- 

0.170 

1 

0.029 

0.1701 

0.177 

VI 

- 

0.336± 

0.043 

0.1901 

0.265 

0.1861 

0.156 

- 

- 

0.50 

0.1001 

0.070 

vn 

- 

0.106± 

0.073 

0.2922 

10.239 

0.4761 

0.088 

0.333 

- 

0.389 

1 

0.096 

0.1961 

0.130 

vm 

0.500 

0.438± 

0.285 

0.75 

0.4421 

0.182 

0.4271 

0.269 

- 

- 

0.3941 

0.237 

In  entries  without  Standard  Deviation  there  is  only  one  value.  Crossed  out 
entries,  contain  Average  and  Standard  deviation  of  only  two  data  values 
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C.  CONCLUSIONS  ON  CHAPTER  ID 


In  this  chi^ter  the  failure  region  analysis  on  the  results  of  the 
CONFLICT  experiment  identifies  the  logical  relation  of  the  faults  responsible 
for  the  observed  failure  regions. 

The  testing  of  the  taxonomical  clustering  hypothesis,  section  3.3  to  3.S, 
implies  the  observed  fault  distribution  is  not  uniform.  The  control  logic,  case 
selection,  and  control  state  faults  appear  at  a  rate  about  fourteen  times  the 
average  fault  ratio  in  the  program  ^ile  processing  and  initialization  fault 
rate  is  about  30%  of  the  average.  Therefore  control  logic  faults  and 
corresponding  failure  regions  actually  exhibit  taxonomical  clustering 
behavior.  This,  however,  does  not  correlate  with  the  structural  clustering  of 
failure  regions  observed  by  Ginn  (1991)  on  the  same  set  of  data.  This  can 
be  justified  by  the  fact  that  the  latter  depends  strongly  on  the  control  and 
data  flow  structure  of  the  program,  with  respect  to  the  corresponding  fault 
location,  while  the  former  depends  on  the  type  of  statement  at  the  fault 
location. 

The  results  of  subsections  3.6  to  3.8  show  that  about  2/3  of  the  faults, 
on  the  avenge,  appear  in  the  functional  modules  of  Attrition,  Communica- 
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tion,  and  Observation  which  include  approximately  50%  of  the  program 
lines.  This  implies  some  mild  tendency  of  the  failure  regions  for  functional 
clustering,  since  some  functional  modules  are  indeed  more  fault-prone  than 
others. 

From  subsection  3.9  we  can  see,  comparing  our  results  with  Ginn's 
(1991),  that  for  all  pairs  of  structurally  correlated  failure  regions,  the 
clustering  metric  is,  on  the  average,  higher  for  functionally  correlated  ones. 
This  is  justified  by  fact  that,  in  a  reasonably  well  structured  program,  faults 
on  the  same  group  of  functional  modules  will  share  a  common  control  and 
data  flow  path  segment  more  often  than  faults  on  different  groups,  therefore 
functional  and  structural  correlation  cannot  be  independent. 
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IV.  GQNCLUSKHMS  AND  SUGGESTIONS  FOR  FURTHER  RE¬ 


SEARCH 

It  has  been  conjectured  in  the  past  that  fault  occurrences  tend  to 
convolve  on  program  locations,  v^ich  implies  that  the  revealing  of  a  fault 
might  indicate  the  existence  of  others  in  close  proximity  of  location. 
However,  the  evidence  so  far  has  been  mostly  anecdotal.  This  thesis,  together 
with  Ginn's  (1991 )  have  been  of  die  first  to  analyze  the  relationships  between 
specific  faults  using  structural  (Giim  1991),  taxonomical  and  functional  (this 
thesis)  criteria.  The  results  of  both  support  the  hypothesis  of  fault  clustering 
and  suggest  methods  for  the  e>q>loitation  of  them  in  software  testing.  This 
chqiter  summarizes  these  results,  in  conjunction  with  previous  work,  and 
points  towards  the  research  questions  which  are  open  to  further  investigation. 

A.  CONCLUSIONS 

This  thesis,  being  a  sequel  to  previous  work  by  Ginn  (1990),  offers 
strong  evidence  that  failure  regions  tend  to  form  clusters,  not  only  wiien 
structural  criteria  are  used,  but  taxonomical  and  fimctional  as  well. 
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The  clustering  criteria  used  in  this  thesis  were  imposed  externally,  since 
both  the  fault  taxonomy  and  the  functional  classification  of  the  faults  have 
been  independent  from  the  experimental  data  structure.  Therefore,  the 
observed  clustering  tendency  of  failure  regions  can  be  characterized  as  global 
as  opposed  to  the  local  clustering  tendency  explored  b  Ginn  (1990).  In  local 
clustering  the  criteria  are  strongly  dependent  upon  the  structure  of  the  data 
at  hand  (Jain  and  Dubes  1988).  Therefore  the  Jaccard  coefficients  used  by 
Ginn  readily  fall  into  this  category. 

The  taxonomical  clustering  behavior  suggests  that  parts  of  the  program 
prone  to  control  logic,  case  selection  and  control  state  faults,  must  be  the 
focus  of  the  testing  effort.  This  implies  that  these  parts  of  a  program  must 
also  be  thoroughly  and  extensively  documented  in  order  to  facilitate  this 
focus  of  effort.  The  CONFLICT  experiment  (Shimeall  1991a,  b)  data 
analysis  suggests  that  decision  points  and  program  control  flow  have  a  higher 
probability  of  fault  occurrence  than  other  locations  of  the  code.  A  good 
testing  or  documentation  mediod,  such  as  decision  tables  etc.,  is  expected  to 
reduce  substantially  the  rate  of  this  type  of  faults. 

The  usefulness  of  the  functional  clustering  behavior  is  that  known  faults 
in  a  program  imply  that  the  probability  of  more,  undiscovered,  faults  in  the 


45 


functional  module  is  higher  than  the  in  rest  of  the  code.  This  provides  e>q>e- 
rimental  evidence  in  support  of  the  anecdotal  conjecture  that  faults  tend  to 
attract  other  foults  (Myers  1979).  It  also  suggests  that  the  distribution  of 
faults  during  the  first  tests  indicates  the  most  fault-prone  functional  modules, 
v^ich  should  be  singled  out  for  additional  testing. 

More  often  than  not  the  testing  effort  is  exponential,  or  of  higher 
complexity,  in  the  length  of  code.  The  ability  to  point  out  the  most  fault- 
prone  modules  or  constructs,  under  the  experimentally-verified  assumption 
of  functional  and  taxonomical  clustering,  represents  a  substantial  reduction 
to  the  required  amount  of  testing. 

The  nature  of  the  cluster  formation,  and  the  correlation  to  Ginns 
structural  clustering,  for  the  two  critoria  used  was  markedly  different.  The 
taxonomical  classification  tended  to  demonstrate  a  clustering  of  type  C  faults 
(and  failure  regions)  in  numbers  one  order  of  magnitude  higher  than  the 
expected  when  a  uniform  fault  per  line  of  code  distribution  was  assumed. 

In  order  to  compare  the  taxonomical  clustering  of  failure  regions  to  the 
structural  clustering  of  the  same  regions,  we  calculated  both  the  average 
Jaccard  coefficient  and  the  average  non-zero  Jaccard  coefficient  (using  Ginn's 
results)  for  every  type  of  possible  taxonomical  clustering  and  every  version 
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of  the  program.  The  results  in  Tables  3.2  and  3.3,  in  Chapter  111,  suggest  that 
the  structural  clustering  of  failure  regions  does  not  depend  on  taxonomical 
clustering.  This  is  attributed  to  the  &ct  that  structural  clustering  depends 
strongly  on  the  control  and  data  flow  structure  of  a  program  with  respect  to  the 
corresponding  &ult  locations  A^e  the  taxonomical  clustering  depencte  mainly  on 
the  type  of  program  statement  at  the  said  locations. 

The  functional  criterion  revealed  a  tendency  of  faults  to  concentrate  on 
cemdn  fiinctiaial  groups  of  modules  (In  die  CONFLICT  case  in  die  Communication, 
Observation  and  Attrition  groups).  A  quite  interesting  result,  in  this  analysis, 
has  been  the  small  scale  clustering  exhibited  by  the  feults,  which  tended  to  occur 
in  high  numbers  within  certain  procedures  (93)  while  the  majority  of  the  examined 
procedures  (446  total,  in  all  eight  versions  of  CONFLICT)  were  faultless  (cf 
Appendix  D  for  a  detailed  functional  fault  type  distribution).  Similar  behavior 
of  faults  has  been  reported  by  Myers  (1979)  but  no  explanation  was  cited.  The 
small-scale  clustering  suggests  that  some  procedures  are  more  complicated, 
therefore  more  fault-prone,  than  others.  In  the  eight  versions  of  the  CONFLICT 
experiment,  the  average  procedure  was  30±20  lines  of  code  in  length  wliile  the 
average  length  of  procedures  with  at  least  one  fault  has  been  42.5±33  lines  and 
with  two  or  more  faults  55±45  lines.  This  result  provides  evidence  in  support  of 
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die  aboveHnentkmed  argument  about  procedure  complexity  and  fault 
clustering  correlation,  de^he  the  inaccuracies  introduced  by  the  fact  diat 
only  fiuilts  widi  a  well-defined  location  were  considered. 

Unlike  the  taxonomical-to-strucniral  clustering  lack  of  correlatum,  in 
testing  the  functional-to-structural  clustering  correlation  in  the  same  as  above 
way  ->  was  found  that  the  average  non-zero  Jaccard  coefficient  is  systemati¬ 
cally  higher  for  functionally  correlated  pairs  than  for  uncorrelated.  This  mild 
correlation  is  explained  by  the  fact  that  in  a  reasonably  well-structured 
program,  fiiults  of  the  same  group  of  functional  modules  will  share  a 
common  control  and  data  flow  path,  will  be  structurally  correlated,  more 
often  than  faults  on  different  groups. 

The  use  of  the  number  of  lines  of  code  (excluding  comments  but  not 
variable  declarations)  in  our  analysis  instead  of  the  HatsUad  length  is  due 
to  the  ease  of  use  of  this  metric  as  well  as  to  unpublished  calculations. 


48 


B.  COMPARISCM^  OF  RESULTS  TO  PREVIOUS  WORK 

The  lesults  of  this  thesis  geneially  support  the  fmdmgs  previous 
researchers  in  the  area  of  relationships  between  fiiuhs.  This  agreement 
suggests  that  the  relationships  may  not  be  specific  to  the  CONFLICT 
software  experiment  but  have  a  more  general  validity  in  large  software 
q)plications. 

Myers  (1979)  postulates  that  functional  clustering  exists  but  provides 
no  further  evidence  or  explaiuition  of  the  basis  for  it  This  thesis,  togedier 
with  the  companion  work  of  Giim  (1990)  makes  a  step  toward  identifying 
the  specific  behavior  of  faults  that  result  in  failure  region  clustering. 

The  eminent  role  of  control  logic,  case  selection  and  control  state  (type 
C)  faults  in  program  testing  has  been  always  enq>hasized  (Beiser  1990, 
Myers  1979),  therefore  the  importance  of  taxonomical  clustering  cannot  be 
easily  overlooked. 

Briliant  et  al  (1990),  analyzing  die  faults  in  a  27-version  software 
experiment,  conclude  that  the  faults  across  independent  versions  not  only  are 
not  independent  but  die  interdependence  is,  in  many  cases,  more  pronounced 
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among  logically  related  (which  includes  taxoncmically  and  functionally 
related)  faults. 

Fkiitfaer  study  of  die  fault  relations  on  the  same  veision  or  across 
versions  is  required  for  establishment  of  the  interaction  mechanisms. 

C.  SUGGESTIONS  FOR  FURTHER  RESEARCH 

While  the  results  of  diis  work  are  promising,  the  experimental 
population  was  small  and  narrowly  focused.  Additionally,  the  programs  were 
written  by  students.  Both  the  method  and  die  results  should  be  validated 
using  a  broad  range  of  professionally-produced  applications. 

One  weakness  of  die  mediod  used  in  diis  diesis  is  that  in  testing  the 
taxonomical  clustering  hypothesis,  a  small  number  of  faults  (about  10%) 
caimot  be  classified  and  therefcne  diey  are  ncrt  included  in  die  analysis.  This 
is  expected  to  introduce  some  minor  inaccuracies  in  the  results. 

Another  weakness  is  die  use  of  seven  fimcdonal  groups  of  modules  in 
the  fimctimial  clustering  analysis.  Movement,  Observation,  Attrition, 
Communication,  Environment,  Restoration  and  Others.  This  coarse  functional 
decompositicm,  imposed  by  die  necessity  of  a  common  base  for  comparison 
for  all  eight  versions  of  CONFLICT,  blurs  the  small  scale  clustering,  hi 
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addition,  the  number  of  lines  per  module  is  quite  a  weak  metric  for  a 
proceduie  complexity  and  cannot  be  used  effectively  as  an  oracle  to  single 
out  the  most  fault-pimie  modules.  However,  this  simple  metric  indicates  that 
it  is  possible  to  use  some  normal  metric  as  a  predictor  of  the  complexity  of 
a  program  unit  Ftirther  research  is  requited  to  develop  metrics  that  can  serve 
as  oracles  for  fault  prone  modules. 

Finally,  the  Jaccard  coefficient,  based  aa  the  number  of  shared 
dimensions  bounding  two  faihite  regions,  is  not  a  very  efficient  measure  of 
stractural  chistering.  A  better  way  to  establish  a  metric  of  two-fault 
correlation  might  be  the  use  of  the  number  of  padis  through  both  their 
respective  locations.  This  mediod  was  not  used  in  this  thesis  since  we  have 
been  focused  on  taxonomical  and  functional  rather  than  structural  clustering. 
Howev^,  <nce  certain  ambiguities,  such  as  distributed  faults,  or  faults 
without  a  specific  location,  such  as  a  missing  fmction  etc.,  are  resolved  the 
proposed  metric  will  directly  translate  die  relationship  between  two  feilure 
regions  into  two  sets  of  code  locations. 
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APPENDIX  A:  NOTATION  AND  SYMBOLS 


•  Df.  Paitition  of  the  input  domahi,  by  path  j 

•  P(|[):  Boolean  expression  equal  to  die  conjunction  of  all  branch 
predicate  constraints  along  path  j.  Therefore  all  elonents  of  Dj  satisfy 
PO). 


•  Bound(G)i  The  boolean  expression  which  defines  faihiie  region  G. 

•  Jd  Symbol  for  exclusive  or  of  boolean  expressions. 

•  The  mapping  of  a  subset,  D',  of  the  input  domain  to  the 
output  range,  by  a  sequence,  SjiSj2..Sj.,  of  program  statements  (this 
sequmce  is  call^  the paA proibiei)^  where  j  is  a  path.  Condition 
evaluations  are  retained  in  the  path  products  for  consistency.  In  fact,  the 
path  products  of  the  two  segments  starting  from  a  decision  point  (such 
as  an  if  C  then  else  constract  or  a  loop  exit  condition  etc)  are  preceded 
with  the  condition  C  and  condition  respectively,  dependiag  upon 
which  part  of  the  decision  Of  or  else)  results  in  dieir  execution. 

^  sequence  of  program  statements,  when  a 
subset  of  dim,  SjjSj4^i..Sjj^,  is  Ae  body  of  a  loop. 

•  SjjSj/ •Sj^m  y  ^  sequence  of  program  statements,  when 
a  subset  of  them,  SySjj^i..Sy^,  is  the  body  of  a  loop,  which  executes  at 
least  once. 

•  S(Gf,Gj^:  The  sinqile  matching  coefficient  for  hiilure  regions  and  Gj. 


52 


t 


s(o,op-^!2pi 

where  Om  stands  for  the  non-bounding  dimensions  of  both  failure 
regions  8^  for  the  composite  dimensions  of  Gj  and  Gj  and  a  for  the 
number  of  all  input  dimensions. 


•  J(G^Gj  ):  The  Jaccaid  coefficient  for  failure  regiims  Gi  and  Gj. 


a-OoQ 


where  aoo  stands  for  the  ncm-bounding  dimensions  of  both  failure 
regions  a,i  for  the  composite  dimensions  of  Gj  and  Gj  and  a  for  all 
input  dimensions. 


•  A->B  or,  equivalently,  :  Logical  implication,  A  implies  B  (  or 
-lAvB). 


•  Identicatty  parUcipating  dUmnsiom:  The  dimensions  that  appear  in 
both  boundaries  of  a  pair  of  failure  regions  in  exactly  die  same  way. 

•  CoiHcidentaUy  Participating  Dimensions:  The  dimensi(xis  diat  appear 
in  both  boundaries  of  a  pair  of  failure  regions  but  not  in  an  identical 
way. 

•  Noidmunding  Dimensions:  The  dimensicms  diat,  for  a  pair  of  failure 
regi(ms,  do  not  appear  in  the  boundaries  of  both  regions.  The  boun¬ 
daries  diese  dimensions  place  on  diese  failure  regions  are  no  more 
restrictive  than  their  entire  range  of  values. 

•  ConqposUe  IHmensums:  Collective  name  for  both  Identical  and 
Coincidental  dimensions 
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•  CMSqmn  StatiHie  (j^):  Statistic  used  to  test  whether  a  significant 
difibience  exists  between  an  observed  and  an  expected  number  of 
objects*  (the  expected  number  of  objects  results  horn  an  assumed 
distribution  ai  Ejects  into  categories).  The  greater  the  chi-square 
statistic  the  lower  the  coqfidenee  that  the  sample  data  follow  the 
assumed  distribution.  If  there  ate  k  categories  (tf  objects,  Oj  is  the 
number  of  dbsarved  and  Ej  the  number  of  expected  objects  in  category 
j,  then  the  statistic  has  k*l  degrees  of  freedom  and  is  equal  to: 


/-I 


‘  Siegel  and  Castellan  1988 
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APPENDIX  B:  TAXONOMICAL  FAULT  TYPE  DISTRIBUTION 
Id  this  Appendix  we  present  histograms  (Figures  B1  to  B8,  in  light 
grey)  of  die  relative  fiequency  of  occuirence  of  jfault  types  in  the  eight 
versions  of  CONFLICT  (Shimeall  1991).  The  hiult  types  in  the  histograms 
are  LOOP,  CONTROL,  PROCESS  and  EXCEPTION  HANDLING.  They 
correspond  to  the  faults  of  type  A,  B,  C  and  D  of  Chapter  m  (In  fact  A.  B, 
C,  D  are  just  abbreviations). 

The  fault  type  fiequency  histograms  are  compared  with  the  expected 
fiequency  histograms  of  a  simple  model  which  assumes  uniform  distribution 
of  faults  (histograms  in  dark  grey).  In  figure  B9  we  present  an  overall 
compariscxi  of  the  observed  to  expected  number  of  faults  ratio,  for  all  eight 
versions.  Tliere  is  a  very  sharp  peak  of  die  ratio  distributions  in  the 
CONTROL  type  of  faults,  which  implies  that  diis  type  of  fault  has  a 
fiequency  of  occurrence  much  greater  dian  the  expected  by  die  assumed 
moctel. 
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Figure  Bl:  Taxonomical  Fault  Frequency  in  Version  I  (light  grey)  vorsus 
random  distribution  of  faults  to  program  locations  (dark  grey). 
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Figure  B.2:  Taxonomical  Fault  Frequency  in  Version  n  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 


Figure  B.3:  Taxonomical  Fault  Frequency  in  Version  HI  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 
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Figure  B.4:  Taxonomical  Fault  Frequency  in  Version  IV  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 


Figure  B.5:  Taxonomical  Fault  Frequency  in  Version  V  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 
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Flfnic  B.6:  Taxonomical  Fault  Frequency  in  Version  VI  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 


Figure  B.7:  Taxc^omical  Fault  Frequency  in  Version  VII  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 
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Figure  B.8:  Taxonomical  Fault  Frequency  in  Version  Vm  (light  grey) 
versus  random  distribution  of  faults  to  program  locations  (dark  grey). 


Figure  B.9:  Actual  number  of  taxonomical  faults  to  expected  number  of 
faults  ratio  for  the  eight  veraons  of  the  program. 
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APPENDIX  C:  SHARED  BOUNDING  DIMENSHKVS  IN  TAXQNQMI- 


CALLY  CLUSTERED  FAILURE  REGIONS 

In  this  Appendix  we  present  a  ctetailed  analysis,  >^ether  taxonomical 
clustering  of  failure  regions  results  in  an  increased  number  of  shared^ 
bounding  dimensions. 

The  results,  are  presented  in  the  tables  Cl  to  C8.  Each  of  the  tables, 
corresponds  to  one  of  the  eight  versions  of  die  CONFLICT  program 
(Shimeall  1991,  1991b).  Each  table  entry,  corresponds  to  a  pair  of  failure 
regions  of  the  same  version.  It  is  noted  that  the  numbering  of  the  failure 
regions  is  not  significant  in  this  analysis.  It  merely  represents  the  order  in 
wliich  the  faults  and  the  corresponding  failure  regions  were  discovered. 

Table  entries  on  the  main  diagonal,  contain  the  logical  cluster  identifier 
(A  to  D),  for  the  corresponding  failure  region  (cf.  Chapter  III).  Each  of  the 
off  diagonal  entries  contains  the  Jaccard  coefficient  for  the  identical 


In  this  analysis,  shared  dimensions  are  the  ones  that  correspond  to  the  same  predi 
cates,  identical. 
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dunensioos  of  the  two  fiEdhue  regions  labeling  the  row  and  column,  and  the 
fault  type  identifier,  in  case  the  regions  belong  to  the  same  logical  cluster. 

For  example,  entiy  (1.2,1.5)  contains  0.100./B  which  means  that  the 
Jaccard  Coefficient  for  these  fiuhire  regions  is  equal  to  0.100.,  and  that  both 
c(»respond  to  faults  of  type  B.  On  die  odier  hand,  enhy  (1.12,1.5)  ccxitains 
0.1S4,  so  the  two  failure  regions  have  a  Jaccard  coefficient  equal  to  0.1S4, 
but  conesp(Hid  to  different  types  of  hmlts  (D  and  B  respectively,  in  this 
case). 
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TABLE  Cl  : 

JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF  VERSION  I 
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0 
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TABLE  Cl  :  JACCARD  COEFFIC3ENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 


TABLE  C.1  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS 

OF  VERSION  I 
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TABLE  Cl  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS 

OF  VERSION  I 
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TABLE  C.1  :  JACCARD  COEFTICIENTS  OF  FAILURE  REGIONS 

OF  VERSION  I 


TABLE  C.1  :  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS 

OF  VERSION  I 
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TABLE  :  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 
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TABLE  CJ  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 
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TABLE  C.2  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 
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TABLE  C.2  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 
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CSB 

OB 

OB 

OB 

OB 

OB 

2.23 

OB 

OB 

OB 

OB 

OB 

OB 

OB 

2M 

OB 

OB 

OB 

OB 

OB 

mm 

OB 

06 

2J9 

OB 

OB 

OB 

OB 

OB 

OB 

OB 

06 
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TABLE  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS 

OF  VERSION  m 


3.18 

0.067 

0.077 

0 

0 

0 

0.091 

0.714 

0A3WC 

0/C 

3J1 

0 

0 

0 

0 

0 

umi 

0 

0 

3.22 

0.125- 

/B 

0.167- 

m 

0/B 

0/B 

0/B 

0.200 

0.063 

0.048 

0 

3J3 

ai82/B 

0200/8 

0.077/B 

om 

0.0833/B 

0I»1 

0.111 

0.040 

0.111 
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33 

D 

m 

mm 

I^QI 

33 

3.10 

3J4 

0.143 

a  167 

0.167 

0 

0.286 

0 

0.071 

0/C 

a250- 

IC 

3J5 

0.125 

a  167 

a250 

0 

0 

■IBfl 

0.067 

0/C 

0/C 

3J6 

O/B 

0/B 

ai2S/8 

0571/8 

0 

0 

0 

0.167 

3JS 

ai2«B 

0/B 

0.100/8 

0222/8 

0.125 

0.063 

0 

0 

332 

0/B 

(VB 

ai«7/B 

(VB 

0/B 

0 

0 

0 

0 

333 

0/B 

0/B 

0.143/B 

(VB 

O/B 

0 

0 

0 

0 

334 

0/B 

0/B 

0.143/8 

O/B 

O/B 

0 

0 

0 

0 

335 

0/B 

0/B 

ai43/B 

O/B 

O/B 

0 

D 

0 

0 

335 

0/B 

0/B 

ai«7/B 

O/B 

(VB 

0 

■■ 

0 

0 

337 

Q/B 

0/B 

0.143/8 

O/B 

0/B 

0 

0 

0 

0 

338 

aiii/B 

0/B 

0/B 

ai23/8 

0 

0.067 

0 

0.167 

339 

0/B 

0/B 

(VB 

ai23/8 

0222/8 

0 

0 

0 

3.48 

0/B 

0/B 

0/B 

m 

0 

0 

0 

0.200  1 

3.41 

aOTl/B 

0/B 

(VB 

ai2S/B 

im^Qi 

0 

0 

0 

3.42 

(VB 

0.111/B 

(VB 

0.126/8 

0 

0 

0 

0.200 

3.43 

a091/B 

0/B 

(VB 

0.167/8 

O/B 

0.143 

0.118 

0.042 

0 

B9 

0/B 

0.083/B 

(VB 

0/B 

0/B 

0.091 

0.770 

0.381 

0 

0/B 

0/B 

(VB 

0/B 

0/)S6/8 

0 

0 

0.529 

0.381 
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3.11 

3.12 

3.13 

3.15 

3.16 

3.17 

3.1S 

3J1 

3J22 

3J3 

3^4 

0 

0 

0/C 

0.10- 

(VC 

0 

0 

0.07- 

7/C 

0 

0 

0.18 

2 

3J5 

0.111 

0.100 

(VC 

0.09- 

1/C 

0.11 

1 

0 

0.08- 

3/C 

0 

0.16 

7 

0.08 

3 

3M 

(VB 

0 

0.091 

0 

Q/B 

0 

0 

g 

0.0- 

83- 

m 

3M 

0.11- 

1/B 

0.11- 

im 

0 

0 

0.11 

1 

(VB 

0.063 

0 

0.11- 

1/B 

0.0- 

83- 

/B 

3J2 

(VB 

(VB 

0 

0.100 

0 

0/B 

0 

m 

Q/B 

Q/B 

3J3 

Q/B 

(VB 

0 

0.091 

0 

Q/B 

0 

0.25 

0 

Q/B 

Q/B 

3J4 

(VB 

(VB 

0 

0.091 

0 

Q/B 

0 

0.25 

0 

Q/B 

Q/B 

3J5 

0/B 

(VB 

0 

0.091 

0 

Q/B 

0 

0.25 

0 

Q/B 

Q/B 
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3.11 

3.12 

3.13 

3.15 

3.16 

3.17 

3.18 

3.21 

3.22 

3.23 

(VB 

m 

B 

0.09 

1 

0 

(VB 

0 

0.25 

0 

O/B 

3J7 

(VB 

(VB 

B 

0.09 

1 

0 

(VB 

0 

B 

(VB 

B 

3J8 

(VB 

(VB 

B 

B 

0 

(VB 

0.063 

0 

(VB 

0.08- 

3/B 

3M 

(VB 

B 

B 

0 

(VB 

0 

0 

(VB 

ao6- 

3/B 

3A9 

(VB 

B 

0 

0 

(VB 

0 

0 

(VB 

0.08- 

3/B 

3At 

(VB 

(VB 

B 

0 

0 

(VB 

0 

0 

(VB 

0.08- 

3/B 

3A2 

(VB 

noil 

0 

0 

0 

Q/B 

0 

0 

(VB 

0.08- 

3/B 

3A3 

ao8- 

3/B 

0.07- 

7/B 

0 

0 

0.08 

3 

(VB 

■ 

0 

0.111- 

m 

0.06- 

7/B 

Q 

0.06- 

7/B 

0.06- 

3/B 

0 

0 

0.06 

7 

(VB 

0.910 

0 

0.077- 

/B 

0.05- 

6/B 

I  3.45 

0.05- 

(VB 

0 

0 

0 

O/B 

0.471 

0 

(VB 

O/B 
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3J4 

335 

3J6 

335 

3J2 

3J3 

334 

335 

3J4 

C 

- 

- 

- 

- 

- 

- 

- 

3J5 

0 

C 

- 

- 

- 

- 

- 

- 

3J6 

0.333 

B 

- 

- 

- 

- 

- 

3M 

0.12S 

0.111 

0.333- 

/B 

B 

- 

- 

- 

- 

332 

0 

0.167/B 

0/B 

B 

- 

- 

- 

333 

0 

0.143/B 

(VB 

B 

- 

- 

334 

0 

0 

ai43/B 

(VB 

OJOO/B 

0.4001/B 

B 

- 

335 

0 

0 

ai43/B 

(VB 

0.40Q/B 

0.40iyB 

B 

334 

0 

0 

0.143/B 

(VB 

0.40Q/B 

0.400/B 

0.400/B 

331 

0 

0 

0.143/B 

(VB 

OJOO/B 

0.40Q/B 

0.40(VB 

0.400/B 

335 

0.286 

0 

OJ75/B 

03(VB 

(VB 

O/B 

(VB 

(VB 

3J9 

0.143 

0 

02S(VB 

o.io(yB 

(VB 

(VB 

(VB 

O/B 

3.4« 

0.143 

0 

02S(VB 

0.1<XVB 

MM 

O/B 

K9 

O/B 

3.41 

0.143 

02S0IB 

o.io(yB 

0/B 

1^9 

(VB 

O/B 

3.42 

0.143 

0 

02SO/B 

0.100/B 

(VB 

Q/B 

Q/B 

O/B 

3.43 

0 

0.125 

Q/B 

0.0S3/B 

K9 

(VB 

O/B 

O/B 

3.44 

0.077 

0.083 

0/B 

0.063/B 

0/B 

0/B 

19 

3.45 

0.059 

0 

0.059th 

0.0S/B 

O/B 

(VB 

O/B  1 
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TABLE  C.4  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS 


OF  VERSION  IV 


■ 

4.1 

44 

4.3 

4.4 

44 

4.7 

4.8 

4.9 

4.19 

4.11 

4.13 

4.1 

C 

- 

- 

- 

. 

- 

. 

- 

. 

- 

- 

4.3 

0 

• 

- 

- 

- 

- 

- 

- 

- 

- 

- 

4J 

otc 

0.118 

C 

- 

- 

- 

- 

- 

- 

- 

- 

4.4 

<vc 

B 

OJMSt 

C 

B 

B 

- 

- 

- 

B 

- 

B 

4J 

0 

0.071 

1 

- 

- 

- 

- 

- 

. 

4.7 

B 

(VB 

(U199 

0j071 

0.167/ 

B 

B 

- 

- 

- 

- 

- 

4.S 

B 

0.1«7 

0X»9 

m 

am/ 

B 

B 

B 

- 

- 

- 

4.9 

(¥C 

B 

ojoat 

c 

<wm/ 

c 

0430 

ai67 

0.12S 

B 

- 

- 

- 

4.19 

B 

(VB 

aos6 

0i)63 

m 

om/ 

B 

0.123 

B 

• 

• 

411 

<vc 

0.QS3 

0M3/ 

c 

(VC 

B 

B 

B 

(VC 

B 

B 

- 

4.13 

<vc 

B 

0.063/ 

c 

0414/ 

C 

0.167 

0.125 

aioo 

m 

B 

(VC 

c 

4.13 

0 

0 

0 

0 

0 

0 

0 

0 

n 

0 

0 

4.14 

<vc 

0 

VC 

VC 

0 

0 

0 

VC 

B 

VC 

(VC 

4.19 

0.300 

(VB 

0.091 

B 

B 

(VB 

0414/ 

B 

B 

B 

B 

4.14 

B 

0.071/ 

B 

B 

0.188 

(VB 

(VB 

VB 

B 

(VB 

B 

0.077 

4.13 

VC 

0.100 

VC 

VC 

0 

0 

0 

(VC 

0 

VC 

(VC 

449 

0 

0.100 

0 

0 

0 

0 

0 

0 

0 

0 

B 

441 

0 

0.071 

0 

0 

0 

0 

0 

0 

0 

0 

B 

443 

0 

(VB 

0 

0 

IQI 

VB 

B9 

0 

id 

0 

0 

444 

<vc 

0.100 

VC 

0 

0 

0 

0 

(VC 

0 

VC 

VC 

444 

<vc 

0.083 

VC 

0 

0 

0 

0 

(VC 

0 

VC 

VC 
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TABLE  C5:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


VERSION  V 


r- 

B| 

53 

5J 

f 

5.5 

5.6 

5.8 

wm 

B 

- 

- 

- 

- 

- 

- 

- 

- 

5J 

(hB 

B 

- 

- 

- 

- 

- 

- 

- 

5J 

(^B 

(KB 

B 

- 

- 

- 

- 

- 

- 

f 

(KB 

OOSISB 

■a 

B 

- 

- 

- 

- 

- 

aos9 

0375 

C 

- 

- 

- 

- 

SJi 

(KB 

(KB 

0.1ff7NB 

(KB 

mm 

- 

- 

- 

■a 

(KB 

(K£ 

(KB 

(KB 

0 

(KB 

B 

- 

- 

5^ 

am 

0 

a059 

0.067VC 

0 

0 

C 

- 

53 

(KB 

OilSlSB 

(KB 

(KB 

0 

(KB 

0 

B 

5.1t 

Kl 

rnim 

OB 

(KB 

0 

(KB 

ESI 

IESE 

0.143\B  1 

5.11 

rnim 

(KB 

0 

■a 

^a 

0 

■ffB 

5.12 

(KB 

(KB 

(KB 

(KB 

0 

(KB 

0 

Bi 

5.13 

0 

ao50 

0 

ao83 

0.100 

0 

0 

aose 

0 

5.15 

(KB 

(KB 

(KB 

(KB 

0 

(KB 

(KB 

0 

(KB 

5.16 

(KB 

(KB 

(KB 

(KB 

0 

Ison 

(KB 

0 

(KB 

5.17 

(KB 

(KB 

(KB 

(KB 

0 

(KB 

E9I 

lESIII 

(KB 

5.18 

(KB 

(KB 

(KB 

(KB 

0 

1^1 

(KB 

0 

E9 

5.19 

(KB 

(KB 

(KB 

(KB 

0 

ESI 

BE 

0 

(KB  1 
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TABLE  CS:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  V 


5^  5.1t  5.11  5.12  5.13  5.15  5.16  5.1  5.18  5.19 


5Jf 

(VB 

D 

(VB 

EEHSII!!] 

(VB 

5J1 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

B] 

(VB 

(VB 

5.22 

(VB 

(VB 

(VB 

a 

(VB 

1^1110311^01 

(VB 

5J3 

■a 

(VB 

(VB 

(VB 

a 

(VB 

(VB 

(VB 

(VB 

(VB 

5J4 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

5J5 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

5J6 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

5J7 

(VB 

(VB 

(VB 

(VB 

(VB 

la 

(VB 

(VB 

SM 

i^9E9 

la 

0 

(VB 

(VB 

(VB 

5J8 

H^i 

(VB 

l!9 

(VB 

0 

la 

(VB 

(VB 

(VB 

(VB 

5J1 

0 

0 

0 

0 

0 

0.130 

0J61 

0 

a 

0 

5J2 

BS 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

5J3 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

5J4 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

5J5 

(VB 

(VB 

(VB 

(VB 

0 

0.040/ 

B 

0.040/ 

B 

(VB 

(VB 

(VB 

5J7 

0 

0 

0 

0 

0 

0 

0 

0 

0 

5.39 

0 

0 

0 

0 

0 

0.083 

0.083 

0 

0 

0 

5.40 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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5.19  5.20 


5.20  (VB 


5J1  0.444/  (VB 

B 


5J2  Q/B  0.286/B  (VB 


0.444/ 

B 


5J7  0.444/ 

B 


5M  (VB 


5.30  (VB 


5J1 


5.32  (VB 


533  0/B 


535  0/B 


(VB 

0.444/ 

B 

Q/B 

B 

0.286/B 

(VB 

0.286/ 

B 

Q/B 

(VB 

0.444/ 

B 

O/B 

Q.444/ 

B 

0.286/B 

0/B 

0.286/ 

B 

0/B 

0/B 

0.444/ 

B 

0/B 

0.444/ 

B 

O/B 

O/B 

0/B 

0/B 

0/B 

(VB 

0/B 

0/B 

0 

0 

0 

0 

0/B 

0/B 

0/B 

0/B 

(VB 

O/B 

Q/B 

Q/B 

0 

0 

Q 

Q 

(VB 

0/B 

Q/B 

Q/B 

0 

0 

Q 

Q 

0 

0 

0 

Q 

0 

0 

Q 

Q 

0.286/  (VB 
B 


0/B  0.444 

/B 


(VB  (VB 


(VB  0/B 


(VB  (VB 


(VB  0/B 


0/B  0/B 
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538 

5J6 

5.31 

5.32 

5J3 

5J4 

5J5 

5J7 

539 

5M 

B 

• 

- 

- 

- 

- 

- 

- 

- 

5Jt 

(VB 

B 

- 

- 

- 

- 

- 

- 

- 

5J1 

0 

C 

- 

■  - 

- 

- 

- 

- 

532 

(VB 

(VB 

B 

- 

- 

- 

- 

- 

533 

mm 

1^1 

(VB 

(VB 

B 

- 

- 

- 

- 

534 

0 

(VC 

0 

B 

- 

- 

- 

535 

(VB 

(VB 

0.(J53/ 

B 

(VB 

(VB 

0.214/ 

B 

B 

- 

- 

531 

0 

0 

0 

0 

0 

0 

0 

- 

- 

5J9 

0 

0 

0.125 

0 

0 

0 

0.063 

0 

- 

5.46 

0 

0 

0 

0 

0 

0 

0 

0 

0.053 
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TABLE  C.6  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


VERSION  VI 


_ 

6.1 

6J 

63 

wm 

63 

■a 

■a 

63 

mm 

C 

- 

- 

- 

- 

- 

- 

- 

0 

B 

- 

- 

- 

- 

- 

- 

<3 

0.357/ 

C 

0 

C 

- 

- 

- 

- 

- 

6.4 

A 

- 

- 

- 

- 

63 

(VC 

0 

o/c 

0 

c 

- 

- 

- 

6.6 

0.077 

0 

0.063 

0 

0 

D 

- 

- 

m 

0.05^ 

C 

0 

0.048/ 

C 

0 

o/c 

0.091 

c 

- 

63 

m 

0.143 

0/C 

0 

o/c 

0.091 

0.500/ 

C 

C 

63 

0.0S9/C 

0 

OMOIC 

0 

o/c 

0.100 

t>3t6IC 

0.28erC 

6.11 

0/C 

0.143 

0/C 

0 

o/c 

0 

O/C 

O/C 

6.12 

0/C 

0 

0/C 

0 

o/c 

(VC 

O/C 

6.13 

0 

(VB 

0 

0 

0 

0 

6.14 

0/C 

0 

0/C 

0 

o/c 

0 

(VC 

o/c 

6.15 

0/C 

0.111 

o/c 

0 

o/c 

o/c 

o/c 

6.16 

0J64/C 

0 

02U/C 

0 

o/c 

0.125 

0.083/C 

0250IC 

6.17 

0.056 

/B 

0.048 

0 

0 

0.083 

0.091 

0.091 

6.18 

0/C 

o/c 

0 

o/c 

0 

O/C 

O/C 
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VERSION  VI 


6.1 

6J, 

6.3 

4.4 

6.5 

mm 

6.7 

6J 

0 

(VB 

0 

0 

im^m 

0 

0 

0 

6J3 

aos6 

(VB 

0.048 

0 

0 

0.091 

0.091 

0.091 

6J4 

(VC 

umi 

(VC 

0 

(VC 

(VC 

(VC 

TABLE  C.6  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  VI 


D 

6.11 

6.12 

6.13 

6.14 

6.1 

5 

B 

B 

i.lt 

<.30 

B 

B 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

6.11 

(VC 

B 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

6.12 

(VC 

(VC 

B 

- 

- 

- 

- 

- 

- 

- 

- 

- 

6.13 

B 

0 

0 

B 

- 

- 

- 

- 

- 

- 

- 

6.14 

(VC 

(VC 

0.039/ 

C 

0 

C 

■ 

a 

■ 

■ 

6.15 

(VC 

0.143/ 

c 

omi 

c 

0 

(VC 

D 

B 

■ 

B 

6.16 

0.07 

lyc 

(VC 

(VC 

0 

(VC 

(VC 

B 

B 

fl 

6.17 

0.06 

7 

0 

0 

(VB 

0 

0 

0.0 

77 

B 

■ 

6.18 

(VC 

(VC 

0.06 

7/C 

0 

0.40 

(VC 

(VC 

0/C 

0 

B 

- 

6M 

0 

0 

B 

(VB 

0 

0 

D 

(VB 

0 

8 

- 

6.23 

0.07 

1 

0 

0 

(VB 

0 

0 

0.0 

77 

0.73/ 

B 

0 

0/ 

B 

- 

6.24 

(VC 

0.071/ 

c 

a 

0 

0.27 

8/C 

0.063 

/c 

(VC 

0 

0.46 

7/C 

0 

0 

B 

TABLE  C.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


VERSION  vn 


r— 

mm 

73 

73 

mm 

73 

7.6 

||Q|[| 

mm 

B 

- 

- 

- 

- 

- 

- 

a083/B 

B 

- 

- 

- 

- 

- 

73 

0.083/B 

asia/B 

B 

- 

- 

- 

- 

7A 

mem 

(VB 

(VB 

B 

- 

- 

- 

73 

a  167 

0 

0 

0.071 

c 

- 

- 

mm 

(VB 

(VB 

(VB 

(VB 

B 

- 

HBB 

(VB 

(VB 

(VB 

(VB 

0.333/B 

B 

73 

(VB 

(VB 

(VB 

0 

0.333/B 

O.SO(VB 

73 

0 

0.333 

0 

0 

0 

7.1f 

mom 

0.231 

0.214 

0 

(VC 

0 

0 

7.11 

0 

HQHI 

0 

0 

0.200 

0.167 

7.1J 

(VB 

(VB 

IHQII 

0 

(VB 

(VB 

7.13 

im^Qiu 

0 

moniii 

0 

0 

7.14 

0 

0 

0 

0.143 

(VC 

0 

0 

7.15 

ao9i 

0.083 

0 

(VC 

0 

0 

7.14 

(VB 

(VB 

IQS 

■a 

0 

(VB 

TABLE  C.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


7J 

mm 

7.1t 

7.11 

7.12 

7.13 

7.14 

7.15 

7.16 

7J 

B 

- 

- 

- 

- 

- 

- 

- 

* 

1.9 

0 

- 

- 

- 

- 

- 

- 

- 

7.1i 

0 

0 

c 

- 

- 

- 

- 

- 

7.11 

a  167 

mi 

D 

- 

- 

- 

- 

7.12 

mm 

HQH 

0 

B 

- 

- 

- 

7.13 

0 

0 

0 

0 

0 

mm 

- 

- 

7.14 

0 

0 

(VC 

0.100 

mQH 

B 

- 

7.15 

0 

0 

(VC 

0 

0 

0 

(VC 

B 

7.16 

(VB 

0 

0 

mm 

0 

0 

B 

B 
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TABLE  C.7:  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


7.1 

7J 

7J 

7.4 

7J 

■ail 

7.17 

0® 

(VB 

(VB 

0 

(VB  1 

7.1g 

(VB 

(VB 

(VB 

(VB 

0 

7.1» 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

0 

(VB 

mrm 

Q/B 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

(VB 

(VB 

0 

0® 

IBSiii 

(VB 

(VB 

(VB 

0 

0® 

imiQi 

0.100 

0.200 

0.273 

0 

0 

0 

iQii 

0 

0 

0 

0 

(VC 

0 

0.077 

0.455 

0.417 

0 

(VC 

0 

■B 

0.048 

0.038 

0.037 

0 

(VC 

0 

IIE9III 

0 

0 

0 

0 

0 

0.143 

0.077 

0.333 

0.400 

0 

(VC 

0 
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TABLE  C.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


B 

lA 

B 

7.18 

7.11 

7.12 

7.13 

H 

7.15 

7.16 

7.17 

(VB 

(VB 

0 

D 

0 

(VB 

0 

0 

0 

0.4(XVB 

7.18 

(VB 

0 

■■ 

0 

(VB 

0 

0 

0 

0.40(VB 

7.19 

■SB 

(VB 

0 

0 

0 

D 

a 

0 

0.40(VB 

7J18 

(VB 

(VB 

0 

0 

0 

m 

0 

0 

0 

0.40(VB 

7J1 

(VB 

(VB 

0 

D 

0 

0 

0 

0 

m 

7.22 

(VB 

(VB 

0 

0 

0 

(VB 

0 

0 

0 

(VB 

7.23 

(VB 

0 

D 

0 

(VB 

0 

0 

0 

(VB 

7J4 

(VB 

(VB 

n 

■i 

0 

(VB 

0 

0 

(VB 

7J5 

(VB 

(VB 

0 

0 

0 

(VB 

0 

0 

0 

(VB 

1X1 

B 

0 

B 

ao7 

7 

(VD 

0 

0 

0 

0.111 

0 

7J8 

0 

0 

0 

(VC 

0 

0 

0 

(VC 

(VC 

0 

7J9 

m 

0 

0231/ 

c 

0 

0 

a 

(VC 

O.0S3/C 

0 

7J2 

0 

mum 

momi 

(VC 

a 

0 

0 

(VC 

O.OSO/C 

0 

7J3 

Ba 

0.12S 

D 

D 

0333/ 

D 

D 

0 

0 

0 

0 

7J5 

0 

0 

0 

0200/ 

c 

0 

0 

0 

(VC 

0.091/ 

C 

0 
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TABLE  C.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


7.17 

7.18 

7.19 

7J0 

7J1 

7.17 

B 

- 

- 

- 

- 

7.18 

0.40(VB 

B 

- 

- 

- 

7.19 

0.40(VB 

a40(yB 

B 

- 

- 

7.29 

a40(VB 

a40Q/B 

Q.4(XyB 

B 

- 

7.21 

Q/B 

Q/B 

Q/B 

Q/B 

B 

7J2 

(VB 

Q/B 

Q/B 

Q/B 

0.571/B 

7.23 

Q/B 

Q/B 

Q/B 

Q/B 

Q.571/B 

7J4 

cm 

(VB 

(VB 

Q/B 

0.571/B 

7J5 

cm 

Q/B 

Q/B 

Q/B 

0.429/B 

7J7 

0 

0 

0 

0 

0 

7J8 

0 

Q 

0 

Q 

0 

7J9 

0 

0 

0 

0 

0 

7.32 

0 

Q 

0 

Q 

0 

7J3 

0 

0 

Q 

0 

0 

7J5 

0 

Q 

0 

Q 

0 
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TABLE  C.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


122 

7.23 

124 

125 

121 

122 

129 

7.32 

7.3 

3 

125 

102 

B 

- 

- 

- 

- 

- 

* 

- 

- 

• 

12Z 

a57i/ 

B 

B 

- 

- 

- 

- 

- 

- 

- 

- 

12A 

0.571/ 

B 

0.571/ 

B 

B 

- 

- 

- 

- 

- 

- 

- 

12S 

0.429/ 

B 

0.429/ 

B 

0.429/ 

B 

B 

- 

- 

- 

- 

- 

- 

121 

0 

0 

0 

0 

imii^n 

- 

- 

- 

- 

- 

12S 

0 

0 

0 

0 

0 

B 

B 

- 

- 

- 

129 

0 

0 

0 

0 

0.182 

0/C 

B 

- 

- 

- 

122 

0 

0 

0 

0 

0.125 

c/c 

ojoyji 

c 

C 

a 

- 

123 

0 

0 

0 

0 

0/D 

0 

0 

0 

D 

- 

125 

0 

0 

0 

0 

0.200 

0/C 

0333f 

C 

0.034/ 

C 

0 

C 
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TABLE  C.8:  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vra 
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TABLE  C.8:  JACCARD  COEPnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vm 


TABLE  C.8:  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vm 


8.1 

83 

83 

8.4 

83 

8.6 

8.7 

83 

8.9 

8J7 

0 

0 

0 

0 

0 

m 

0 

0 

0 

8^8 

0 

0 

0 

0 

0 

(VB 

0 

0 

0 

8J9 

0 

0 

0 

0 

0 

om 

0 

0 

0 

8J0 

0 

0 

0 

0 

0 

(VB 

0 

0 

0 

8J1 

0 

0 

0 

0 

0 

O/B 

0 

0 

0 

832 

0 

0 

0 

0 

0 

O/B 

0 

0 

0 

833 

0 

0 

0 

0 

0 

(VB 

0 

0 

0 

834 

0 

0 

0 

0 

0 

Q/B 

0 

0 

0 

836 

0J33/D 

(VD 

(VD 

(VD 

0 

0 

0 

0.143/D 

0.091 

837 

0.60(VD 

(VD 

(VD 

(VD 

0 

0 

0 

0333/D 

0.143 

838 

0.600 

0 

0 

0 

0 

0/B 

0 

0.333 

0.143 

8.40 

0.500 

0 

0 

C 

0 

(VB 

0 

0.250 

0.125 

8.41 

0.500 

0 

0 

0 

0 

(VB 

0 

0.333 

0.143 

8.42 

0 

0 

0.500 

0.143 

■El 

0.S0IVB 

0.333 

0 

0 

8.44 

0.333 

0 

0 

0 

0 

(VB 

0.167 

0.100 

8.45 

0.200 

1.00 

0.333 

0.125 

0.500 

0333/B 

0.250 

imi 

0 

8.46 

0.333 

0 

0 

0 

0 

(VB 

0.333 

0.333 

0.200 

8.48 

0.500 

0 

0.500 

0.143 

1.00 

O.S0(VB 

0 

0 

0 

830 

0 

0 

0 

0 

0 

(VB 

0 

0 

0 
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TABLE  C.8:  MCCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vm 


8.11 

8.12 

8.13 

8.14 

8.15 

8.18 

8.19 

S33 

8J5 

8.26 

8J7 

(VB 

O/B 

(VB 

0 

(VB 

(VB 

0/B 

O/B 

0300/B 

0300/B 

0/B 

O/B 

0 

O/B 

(VB 

(VB 

O/B 

ojserm 

0.667/B 

8J9 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

OMO/B 

0.40Q/B 

8J0 

(VB 

(VB 

0 

(VB 

(VB 

(VB 

O/B 

mm 

(VB 

8J1 

O/B 

(VB 

O/B 

um 

(VB 

O/B 

O/B 

0/B 

8^2 

(VB 

O/B 

Q/B 

a 

(VB 

(VB 

(VB 

O/B 

O/B 

0/B 

8J3 

0/B 

O/B 

(VB 

■■ 

(VB 

O/B 

(VB 

(VB 

8J4 

0/B 

O/B 

O/B 

0 

(VB 

O/B 

O/B 

O/B 

8J6 

0 

0 

0.2S0 

0333/ 

D 

0371 

aioo 

0300 

0 

0 

0 

8J7 

0 

0 

0.50 

0321/ 

D 

1.00 

ai67 

0333 

0 

0 

0 

8^8 

(VB 

O/B 

0.S001 

B 

0322 

0.167/ 

B 

033/ 

B 

(VB 

O/B 

(VB 

8.48 

(VB 

(VB 

a40(v 

B 

0.30 

0 

liXVB 

0.143/ 

B 

0.429 

/B 

O/B 

(VB 

(VB 

8.41 

(VB 

(VB 

OSKV 

B 

0.33 

3 

(VB 

0.167/ 

B 

0300 

/B 

(VB 

(VB 

(VB 

8.42 

0333/B 

O/B 

0/B 

0 

0/B 

0/B 

(VB 

0300 

/B 

(VB 

(VB 

8.44 

(VB 

(VB 

0386/ 

B 

0.25 

0 

0j6SJf 

B 

0.111/ 

B 

0.429 

/B 

0/B 

(VB 

O/B 

8.45 

0.2S(VB 

O/B 

(VB 

0.12S 

0300/ 

B 

0350/ 

B 

0/B 

033/ 

B 

(VB 

0/B 

8.46 

(VB 

0/B 

0300/ 

B 

0333 

1.00/B 

0/B 

0300 

/B 

0/B 

(VB 

8.48 

0333/B 

O/B 

(VB 

0 

0/B 

(VB 

(VB 

0300 

IB 

0/B 

(VB 

8.58 

O/B 

(VB 

n 

O/B 

0/B 

(VB 

(VB 

0/B 

(VB 
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TABLE  C.8:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vm 


837 

838 

839 

838 

831 

832 

833 

8^7 

B 

• 

- 

> 

- 

- 

- 

SM 

O-SOO/h 

B 

- 

- 

- 

- 

. 

8^9 

0.33VB 

0.40(VB 

B 

- 

- 

- 

- 

8J0 

0/B 

(VB 

0/B 

B 

- 

- 

- 

8^1 

0/B 

(VB 

0/B 

0.444/B 

B 

- 

- 

8J2 

(VB 

wm 

0/B 

0.667/B 

0.50(VB 

B 

- 

8J3 

(VB 

0/B 

0/B 

0.667/B 

O.S(XVB 

0.80(VB 

B 

8J4 

0/B 

0/B 

0.364/B 

0.308/B 

0.40(VB 

0.40(VB 

8J6 

0 

0 

0 

0 

0 

8J7 

0 

0 

0 

0 

0 

8J8 

— 

■91 

■B 

0/B 

0/B 

0/B 

8.48 

(VB 

0/B 

0/B 

0/B 

0/B 

0/B 

0/B 

8.41 

(VB 

■■ 

— 

0/B 

0/B 

0/B 

0/B 

8.42 

(VB 

0/B 

(VB 

0/B 

0/B 

0/B 

0/B 

8.44 

0/B 

0/B 

0/B 

0/B 

0/B 

0/B 

8.45 

0/B 

0/B 

0/B 

0/B 

0/B 

BQI 

Q/B 

8.46 

(VB 

0/B 

0/B 

0/B 

0/B 

0/B 

0/B 

8.48 

HB3II 

mm 

0/B 

0/B 

0/B 

0/B 

0/B 

838 

(VB 

0/B 

0/B 

0/B 

0/B 

0/B 

0/B 
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TABLE  C.8:  JACCARD  COEFT1CIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vm 


934 

936 

8J7 

8J8 

8.48 

8^1 

8.42 

8.44 

8.45 

8.44 

0 

•  ID 


0.429  1. 


0.500  0.750  0.750 

IB 


8^  0/B 


IB  IB 


0/BO  0  0/B  0/B  0/  B 

B 


nm  0J64  0375  0375  0.444  0300  Q/  g 

^  IB  IB  IB  ^ 


«.4  0/B  0350 

5 


0350  IJXX)  0.143  n 

/B  I  /B  IB  IB 


0/B 


0/B  0 


0.083  0 

IB 


0333  0.400  0.400  0300  0.fi00 

IB  IB  IB 


0375  0367 

IB  IB 


0  0/B  (VB  0/  0/  0/ 

B  B  B  ® 


0  0/B  0/B  0/  0/  0/  0/ 

B  B  B  B 
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APPENDIX  D:  FUNCTIONAL  FAULT  TYPE  DISTRIBUTION 


Id  this  Appendix  we  present  (Tables  D.l  to  D.8)  die  relative  firequency 
of  occuirence  of  functional  fault  types  in  die  eight  versions  of  CONFLICT 
(Shimeall  1991b).  The  fault  types  are  classified  in  types  O,  L  IL  m,  ^ 
and  VI  according  to  die  discussion  in  Section  B.7  of  Chapter  m  . 

Some  of  die  faults  included  in  die  failure  region  libraiy  by  Shimeall 
(1991a)  do  not  have  a  specific  locaticm  and,  therefore,  cannot  be  assigned  to 
any  specific  program  module.  However,  it  is  possible  by  dieir  desciiptian  to 
detennine  which  part  of  the  CONFLICT  functicm  each  of  them  affects.  These 
faults  occupy  the  generic  entries  (labeled  as  Others,  movement,.,  etc)  in  bold 
type  diat  precede  each  functional  group  of  program  modules. 

Faults  that  have  been  assigned  to  multiple  functional  groups  contribute 
to  the  fault  count  of  each  group  by  an  appropriate  fiactirm  of  fault  For 
example,  a  fault  distributed  in  three  groups,  will  contribute  to  the  fault  count 
of  each  group  by  1/3. 

The  observed  fault  type  fiequencies  are  compared  with  the  expected 
frequencies  of  a  simple  model  that  assumes  uniform  distribution  of  faults  as 
in  Appendix  B. 
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TABLE  D.1  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  I‘* 


Piocedm  Type 


Conflict 


Ceiling 


Mini 


MinR 


MaxIR 


Outside  Range 


Verifylnpot 


Chedt  PanuDS 


Check  Anny 
Values 


Ptepaie  Output 


Initialization 


Update  Anny 
Values 


Uaes 


Obeenred 

FMdti 


Rdatfve 

Flmpwncy 

Expected 

IMti 

Rctalive 

ncqnency 

8/26 

6.2 

0.24 

llie  fiault  rate  is  26/2352  =  0.01105  faults  per  line  (induding  data 
dedarations  in  the  line  count) 


TABLE  D.1  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  I 


PiocmIbr 

Type 

Ums 

Obscnrcd 

FMtt 

Iclaihrc 

ncqMBcy 

Expected 

EnHs 

Relative 

Fnqaeaej 

Movanm 

I 

K9 

- 

1/26 

1.77 

a068 

BatVelocV 

I 

23 

- 

- 

- 

- 

AltitudeZ 

1 

30 

- 

- 

- 

- 

DistD 

I 

4 

- 

- 

- 

- 

Position 

I 

72 

- 

- 

- 

Movement 

I 

31 

- 

- 

- 

- 

Otaavatkm 

n 

24a 

- 

3/26 

2.65 

0.10 

SiieUstLoc 

n 

14 

- 

- 

- 

- 

HeightH 

n 

20 

- 

- 

- 

- 

HndAngie 

n 

21 

- 

- 

- 

- 

FiistCondi- 

tion 

n 

41 

- 

- 

- 

- 

Second 

Condition 

n 

25 

- 

- 

- 

- 

Visual 

Contrast 

n 

10 

- 

- 

• 

- 

ObservJam 

n 

18 

\XI 

- 

- 

- 

lUid 

Condition 

n 

27 

- 

- 

- 

- 

Otisen^ation 

n 

64 

1.14, 126 

- 

- 

- 
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TABLE  D.1  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  I 


Type 

Unas 

Ohacnrcd 

Faidli 

EelatiTc 

Piequemj 

Ezpeded 

FMti 

RelidiYC 

Fnqaamy 

AUritlMI 

m 

35S 

- 

4/26 

3.92 

ai5i 

Attrition 

m 

10 

1.13 

- 

- 

- 

Set  File 
Upon 
Cooids 

m 

44 

1.13, 1.1&  I.IS 

• 

- 

- 

Assign  LL 
Cooids 

m 

21 

- 

- 

- 

mmm 

m 

46 

- 

- 

- 

Nmn  Killeis 
NK 

m 

B 

- 

- 

- 

Kffleis 

AvsilKA 

m 

39 

- 

- 

- 

Times 

KiUeis 

UsedKU 

m 

40 

• 

- 

- 

Total  Wei^ 
biUse  NW 

m 

51 

130 

- 

- 

CteaiDead 

Squads 

m 

30 

- 

- 

- 

For  Each 
Weap 

m 

46 

- 

- 

- 

don 

IV 

678 

- 

3/26 

7.49 

0.288 

Ciiedt 

ComMsg 

IV 

21 

- 

- 

- 

- 

Send 

Rq»fts 

IV 

47 

- 

- 

- 

- 

TABLE  D.1  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  I 


nocMhm 

Type 

Lines 

Obacnred 

RmHi 

Relative 

Frequency 

Expected 

F^ti 

Relative 

Frequency 

Piia4^  On 
Sent  LL 

IV 

72 

- 

- 

- 

- 

InstanlialB 

CommamAlsg 

IV 

66 

1.2S.  1.19 

- 

- 

- 

BarinMOMBt 

V 

143 

- 

4/26 

1.58 

0.061 

CSwdi  Weattier 

V 

26 

U.  1.4 

- 

- 

- 

TenMove  TM 

V 

25 

1.18. 1.12 

- 

- 

- 

Weattier 

SevFtetorWF 

V 

22 

- 

- 

- 

- 

Weather 

ObaeivWO 

V 

12 

- 

- 

- 

- 

Weather 

MoveWM 

V 

12 

- 

- 

- 

- 

Slope  biiensity 
IS 

V 

24 

- 

- 

- 

- 

Altitude 

InlensitylA 

V 

7 

- 

- 

- 

- 

imensityLoclL 

V 

6 

- 

- 

- 

- 

Location 
bilensity  BI 

V 

9 

- 

- 

- 

- 
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TABLE  D.l  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  I 


Aoccdare 

Type 

Lines 

OlMMved 

Rralli 

Relative 

Freqnency 

Expected 

Faults 

Reireive 

Frequency 

Rctlomtioii 

VI 

21S 

- 

3/26 

Z38 

0.092 

Squad  Alive 

VI 

4 

1.9 

- 

- 

- 

Bat  Alive 

VI 

16 

- 

- 

- 

- 

Restoiadon 

VI 

8 

- 

- 

- 

- 

TotalRestore 

dCasualtie^ 

F 

VI 

36 

• 

- 

- 

- 

Coefficient 

VI 

28 

- 

- 

- 

- 

MunSquads 

Restoring 

NF 

VI 

19 

• 

- 

- 

- 

ReatoR 

SuppAmffiS 

VI 

23 

- 

- 

- 

- 

Restore 

Factoffi 

VI 

54 

1.5. 1.2S 

- 

- 

- 

CalcEndmfi 

VI 

27 

- 

- 

- 
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TABLE  D.2  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  n“ 


Pioccdm  Type 


Olittn 


Conflict 


MinReal 


Minlnt 


Mu 


Maxim 


Roof 


Floor 


OlMCfvcd 

nmlti 


Relative 

Ftcqaency 

Expected 

Eaelte 

Relative 

Ficqeeocy 

0/26 

3.11 

0.119 

Velocity 


The  fault  rate  is  26/1540  s  0.0169  faults  per  line  (including  data 
declarations  in  the  line  count) 
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TABLE  D.2  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES 

VERSION  n 


Procedwe 

Tjpe 

Uses 

Obscnrcd 

nnlta 

Relative 

ncqaency 

Expected 

EanltB 

Relative 

Ficqaency 

Xmove 

I 

9 

- 

- 

- 

- 

Ymove 

I 

9 

- 

- 

* 

- 

Movement 

I 

37 

- 

- 

- 

- 

Posinit 

I 

14 

- 

- 

- 

- 

Movebiit 

I 

11 

- 

- 

- 

MoveOat 

I 

34 

- 

- 

- 

- 

Otaervatlon 

n 

227 

Z1S.126 

5/26 

3.832 

0.147 

Angle  Big 
Enough 

n 

60 

X7.18 

- 

- 

- 

Slope 

n 

4 

- 

- 

- 

- 

Can  J  Seek 

n 

9 

- 

- 

- 

- 

Observatian 

n 

30 

- 

- 

- 

- 

Calc 

Contrast 

n 

24 

- 

- 

- 

- 

■fW 

n 

5 

WKsm 

- 

- 

- 

No 

Obstacles 

n 

20 

■ 

- 

- 

- 

NotObs 

Jammed 

n 

20 

- 

- 

- 

- 

OJamming 

n 

25 

- 

- 

- 

- 

Obdidt 

n 

10 

- 

- 

- 

- 
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TABLE  D.  2  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES 

IN  VERSION  n 


PiDCMbue  Type 


Attrition 


Attrition 


Attriflnflict 


WeapcNis 


HieCooiti 


Suffering 


Attritinit 


AttritOut 


Comraiinic 

ation 


Communic 

ation 


Update 

Comm 


AddToQ 


Create 

Reports 


Create 

Commands 


PuU 

PitnnQ 


Relay 

Messages 


Consume 

Reports 


Obscnred 

Faalls 


Relative 

Frequency 

Expected 

Fantts 

Relative 

Frequency 

0/26 

4.85 

0.186 

Z16^17.  XIS. 

X19. 120, 
121, 121 123 


14,125 


14/26 


0.240 
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TABLE  D.  2  FAULT  DBTIUBUTION  TO  FUNCTIONAL  MODULES 

IN  VERSION  n 


Procedare 

Type 

Lines 

Relative 

Ftoqnency 

Expected 

Fanils 

Relative 

Frequency 

Consume 

Commands 

IV 

51 

15 

- 

• 

- 

Commlnit 

TV 

15 

- 

- 

- 

- 

Enviranmc 

Bt 

V 

141 

- 

3/26 

2.38 

0.092 

Dist 

V 

4 

- 

- 

- 

- 

Alt 

V 

25 

130,2.6.231 

- 

- 

- 

Podntens 

V 

31 

- 

- 

- 

- 

WTotal 

V 

27 

- 

- 

- 

- 

WMove 

V 

21 

- 

- 

- 

• 

WObs 

V 

21 

- 

- 

- 

- 

Height 

V 

12 

- 

- 

- 

- 

Restoration 

VI 

114 

1/26 

1.50 

0.058 

Restoration 

VI 

89 

- 

- 

- 

- 
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TABLE  D,3  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 


VERSION 


noceduc 

T^pe 

LiMS 

Ohsafvcd 

Fadti 

Relative 

neqaenej 

Expected 

FiaaltB 

Relative 

neqaency 

0 

312 

- 

7.5/40 

10.4 

0.26 

Conflict 

0 

81 

3.10. 3.17, 
301.3.^ 

- 

- 

- 

max 

0 

5 

- 

- 

- 

- 

min 

0 

4 

- 

- 

- 

- 

HESiH 

0 

9 

- 

- 

- 

- 

Change 

o 

6 

- 

- 

- 

- 

Output 

0 

64 

- 

- 

- 

- 

DataUpdaie 

o 

75 

3.15 

- 

- 

- 

Peifoim 

Passive 

Functions 

0 

5 

- 

- 

- 

- 

InitVals 

0 

63 

3.5,3.24.3.36 

- 

- 

- 

Movcnent 

I 

94 

- 

2/40 

3.13 

0.078 

Bafl^tion 

I 

67 

3.25.3.6 

- 

- 

- 

positioning 

I 

8 

- 

- 

- 

- 

Move 

I 

19 

- 

- 

- 

- 

The  fault  rate  is  40/1200  s  0.00333  faults  per  line  (induding  data 
dedarations  in  the  line  count) 
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TABLE  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES 

VERSION  m 


Piocednc  Type 


SuhAngle 


si^ng 


Summation 


obaeive 
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TABLE  D.3  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES 

VERSION  m 
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TABLE  D.4  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  IV** 


Olten 


Conflict 


bitialize 


Piep  Output 


Movement 


Moiveiiieiit 


CalcVdoclty 


MTemin 


Podtlan  Squads 


Observation 


Obseivatian 


Relative 

ncqnency 

Expected 

FbnHi 

Relative 

Fteqaency 

0/23 

3.124 

0.136 

blit  Visual 


ValidCHneivatio 

n 


Angle 


Line 


Obs  Contrast 


Location  List 


Visual  Contrast 


^  Hie  fault  rate  is  23/1995s0.01153  faults  per  line  (induding  data 
dedarations  in  the  line  count) 
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TABLE  D.4  FAULT  DKTRffiUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  IV 
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TABLE  D.4  FAULT  DlSTRraUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  IV 
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TABLE  D.5  FAULT  DISTRIBimON  TO  FUNCTIONAL  MODULES  IN 

VERSION  V” 


Hie  fault  rate  is  40/1544  s:0.026  faults  per  line  (including  data 
dedarations  in  the  line  count) 
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TABLE  D.S  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  V 
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TABLE  D.6  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 


“  Hie  fiault  rate  is  20/2219s0.0090  faults  per  line  (induding  data 
dedarations  in  the  line  count) 
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TABLE  D.6  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  VI 


Pnccdnc  I  T>pe 


Movemm 


Update 

Badalian 

VdocHy 


AlignSquads 


Movement 


Move 

Battalion 


Obaovatioa 


Create 

LOSList 


Obaervatian 


GaiObsList 


CHneivable 


AngleSub 

Greater 


Updat 

LOSUst 


LOSQear 


OntisK^ 


Location 

Intensity 


Obs  lamming 


SumObs 

ToNextBatt 


Observed 

Fbalts 


Reldive 

Flcqwncy 

Expected 

Fanils 

Reldive 

neqnen 

cy 

(V20 

0.86S 

0.043 
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TABLE  D.6  FAULT  DlSTiUBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  VI 
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TABLE  D.6  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  VI 


Type 

Linaa 

NewNum 

Send 

IV 

28 

Send 

P/WMBMld 

rv 

29 

ReoeiveConi- 

mnnicalians 

IV 

11 

Ffaid  Receiv¬ 
ing  Delay 

IV 

55 

Receive 

Rqxnts 

IV 

56 

Receive 

Commands 

IV 

57 

Update 

NumVais 

IV 

30 

PiDcessCom- 

municadons 

IV 

11 

Handle 

Queuing 

rv 

8 

Queue 

Rqxnts 

rv 

38 

Rnd  Queue 
Report 

rv 

19 

Queue 

Commands 

IV 

38 

Rnd  Queue 
Spot 

IV 

19 

Processing 

Delay 


IV 


24 


TABLE  D.6  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  VI 


Observed  Relative 
Fwlti  Fieqaencj 


Expected  Relative 

I^olts  Flcquea 

cy 


0.079 


TABLE  D.6  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  VI 


Piocedare 

Type 

Lines 

Obscived 

FMdte 

Relative 

Ftcqaency 

Expected 

Fralts 

Relative 

Freqnency 

Height 

V 

28 

- 

- 

- 

- 

Update 

Weadier 

V 

7 

- 

- 

- 

- 

Update 

Present 

Events 

V 

41 

- 

- 

- 

- 

Add  New 
Events 

V 

24 

- 

- 

- 

- 

Weather 

Severity 

V 

42 

- 

- 

- 

- 

WEOn 

Movement 

V 

25 

- 

- 

- 

- 

Rcstonlion 

VI 

114 

- 

0/20 

1.027 

0.051 

Restoration 

VI 

17 

- 

- 

- 

- 

NewNum 

Rxeis 

VI 

24 

- 

- 

- 

- 

Appoftiai 

Rxing 

VI 

53 

- 

- 

- 

- 

Remove 

Destroyed 

Squa^ 

VI 

20 

- 

- 

- 

- 

125 


TABLE  D.7  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vn“  ” 


Otamed 

nmUs 


Relative 

Ftcqaenc 

y 

Expected 

nidts 

Relative 

ncqaen 

cj 

4.5/31 

10.385 

0.335 

The  fault  rate  is  31/1800a0.0172  faults  per  line  (induding  data 
dedarations  in  the  line  count) 

In  this  Version,  certain  sub  procedures  are  dedared  in  more  than  one 
locations 
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TABLE  D7.  FAULT  DlSTRmUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vn 


PncMtanc  Type  Lines 


Obsenred  Relative 

FbaHs  Pkcqaency 


Expected  Relative 
Faolts  Ftcqaency 
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TABLE  D.7  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vn 


TABLE  D7.  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vn 


OlMMVcd  Relative 

Fnlli  ncqeeBcv 


Expected  Relative 

Fealli  Ftcqaency 


TABLE  D.7  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vn 


ItocMim 

Type 

fiwff 

Obawrcd 

Iteiti 

Relative 

Ftcquency 

Expected 

FauHi 

Relative 

Ptcqaeacy 

WXPosition 

mm 

5 

- 

- 

- 

- 

WYPositiQn 

V 

5 

- 

- 

- 

- 

V 

10 

- 

- 

- 

- 

WObsenre 

V 

13 

- 

- 

- 

- 

W 

Movoneitt 

V 

12 

- 

- 

- 

- 

TenEffect 

mm 

15 

- 

- 

- 

- 

V 

8 

- 

- 

- 

- 

RcftonlioB 

VI 

97 

7.16 

2/31 

2.0 

0.065 

DeitaHxSup 

VI 

13 

- 

- 

- 

- 

Set 

Restoimdon 

VI 

23 

7.12 

- 

- 

- 

duBige 
Squad  Data 

VI 

61 

- 

- 

• 

- 
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TABLE  DM  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vm** 


Type 

Uaea 

mmm 

■SB 

Rdalivc 

neqaaecy 

Expected 

Fterili 

Edadvc 

PteqMBcy 

(Mhm 

o 

258 

- 

0/41 

7.778 

0.190 

CoofUct 

0 

93 

- 

- 

- 

- 

UpdaieU 

0 

14 

- 

- 

- 

- 

SeiKU 

0 

15 

• 

- 

- 

- 

Init 

VaiisUes 

o 

47 

- 

- 

- 

- 

Lengdi  Of 
List 

o 

11 

- 

- 

- 

- 

UpdaleVais 

0 

53 

- 

- 

- 

- 

Update 

Nums 

o 

4 

- 

- 

- 

- 

Ptepaie 

Ouqmt 

o 

12 

- 

- 

- 

- 

Cefling 

o 

9 

- 

- 

- 

- 

MovcbmbI 

I 

138 

8.18.  8JS 

4/41 

4.160 

0.101 

SquadPos 

I 

54 

12 

- 

- 

- 

Vdocity 

I 

25 

• 

- 

- 

- 

Movanent 

I 

14 

- 

- 

- 

- 

The  fault  rate  is  41/1366  =0.030  faults  per  line  (induding  data 
dedaratians  in  the  line  count) 
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TABLE  DJ  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vm 


32 


TABLE  FAULT  DBTRDUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vm 


Type 

Ums 

Ohscrred 

Relallvc 

neqMBcy 

Bipacled 

PiMill 

letalfvc 

Fiegasncy 

UpdMBKA 

23 

a.1 

- 

- 

- 

UpdUBKU 

21 

- 

- 

- 

- 

CtearAttadc 

11^311111 

14 

- 

• 

• 

• 

Lists 

SetStalns 

17 

- 

- 

- 

- 

jl 

IV 

H 

IJ7.S^ 

&29.8JQ, 

131,832. 

133,134 

ISMl 

10.793 

0.263 

taseiAlsg 

IV 

41 

15 

- 

- 

Command 

IV 

13 

- 

• 

• 

Msg 

bMBitCom 

IV 

17 

- 

- 

- 

RepoiAf^ 

IV 

7 

- 

• 

- 

biseftflq) 

IV 

29 

• 

- 

- 

Gommunica 

IV 

7 

- 

• 

• 

tion 

Receive 

IV 

31 

. 

• 

. 

Delay 

CalRDelay 

IV 

29 

112,130 

- 

- 

QneDday 

IV 

27 

- 

- 

- 

PotQue 

IV 

39 

13,14,16 

- 

- 

- 

PiDcessQiie 

IV 

28 

- 

- 

- 

- 

PiocessMag 

IV 

39 

- 

- 

- 

- 

MeigeRq) 

IV 

12 

- 

. 

- 

. 
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TABLE  FAULT  DISTRIBUTION  TO  FUNCTIONAL  MODULES  IN 

VERSION  vm 


D 

Type 

Obscnrad 

Mil 

Ralativa 

Fnqftaej 

Expected 

Pterin 

Relteivc 

Fteqaency 

Meij^Com 

M|g 

IV 

39 

1.7 

- 

- 

- 

V 

!!• 

- 

7/41 

3.316 

0.081 

Linear 

Distance 

V 

7 

- 

- 

- 

- 

AUitude 

V 

19 

1.42 

- 

- 

- 

WSevFactcnr 

V 

20 

- 

- 

- 

- 

CalcBI 

a 

24 

IS.  8.10, 
S.11. 8.44 

- 

- 

- 

Tenmin  Effect 

V 

21 

- 

- 

- 

- 

Weather 
Move  Effea 

V 

13 

8.4S 

- 

- 

- 

WObsEfifect 

V 

6 

8.46 

- 

- 

- 

Rutontion 

VI 

82S 

1/41 

1653 

0.065 

UpdaleFS 

VI 

18 

- 

- 

- 

- 

UpdateFF 

VI 

15 

- 

- 

- 

- 

UpdaleE 

VI 

55 

- 

- 

- 

- 
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APreNDlX  E:  SHARED  BOUNDING  DIMENSIONS  IN  FUNCTION¬ 


ALLY  CLUSTERED  FAILURE  REGIONS 

In  this  Appendix  we  present  a  detailed  analysis,  whether  functional 
clustering  of  failure  regions  results  in  an  increased  number  of  shared’^ 
bounding  dimensions. 

The  results,  are  presented  in  die  tables  El  to  E8.  Each  of  the  tables, 
corresponds  to  one  of  the  eight  versions  of  die  CONFLICT  program 
(Shimeall  1991,  1991b).  Each  table  entry,  corresponds  to  a  pair  of  &iluie 
regions  of  the  same  version  as  in  Appendix  C. 

Table  entries  on  the  main  dif^onal,  contain  die  functional  cluster 
identifier  (1,  II,  III,  IV,  V,  VI,  O),  for  the  corresponding  failure  region  (cf 
Chapter  III).  Each  of  the  off  diagonal  entries  contains  the  Jaccard  coefficient 
for  the  identical  dimensions  of  the  two  failure  regions  labeling  the  row  and 
column,  and  the  fault  type  identifier,  in  case  the  regions  belong  to  the  same 
functional  cluster. 


In  this  analysis,  shared  dimensions  are  the  ones  that  correspond  to  the  same 
predicates,  identical. 
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For  exan^le,  ootiy  (1.1, 1.6)  contains  0/0  which  means  that  the  Jaccaid 
Coefficient  these  foiluie  regions  is  equal  to  0,  and  diat  bodi  cmiespaid 
to  &ults  of  type  O.  On  the  odier  hand,  enhy  (1.5,  1.12)  contains  0.1S4,  so 
the  two  failure  regions  have  a  Jaccaid  coefficient  equal  to  0.154,  but 
correspond  to  different  types  of  faults  (VI  and  V  respectively,  in  this  case). 
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TABLE  E.1  :  JACCARD  COEFnClENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 


1.1 

IJ 

1.3 

1.4 

13 

1.6 

IJ 

(VO 

0 

- 

- 

- 

- 

13 

0 

0 

V 

- 

- 

- 

lA 

0 

0 

1.333/V 

V 

- 

- 

13 

0 

0.100 

0 

0 

VI 

- 

lA 

(VO 

(VO 

0 

0 

0 

o 

1.7 

(¥0 

(VO 

0 

0 

0 

(VO 

13 

0 

0.083 

0 

0 

0 

0 

13 

IHQH 

0 

0 

0 

(yvi 

(VO 

1.11 

0.333/0 

(VO 

0 

0 

0 

0 

1.11 

a333/0 

(VO 

0 

0 

0 

0/0 

1.12 

0 

0.0833 

(W 

(W 

0.154 

0 

1.13 

0 

0 

0 

0 

0 

0 

1.14 

0 

0 

0 

0 

0 

0 
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TABLE  E.1  :  JACCARD  COEFTICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 


■SB 

1.8 

1.9 

1.18 

1.11 

1.12 

1.13 

1.7 

o 

- 

- 

- 

- 

- 

- 

lA 

0 

I 

- 

- 

- 

- 

- 

1.9 

0 

0.063 

VI 

- 

- 

- 

- 

1.19 

Q/O 

0 

0 

0 

- 

- 

- 

1.11 

0  /O 

0 

0.380  /0 

uniQm 

- 

- 

1.12 

0 

0 

0 

0 

mom 

V 

- 

1.13 

0 

0.182 

0.083 

0 

0 

0 

1.14 

0 

0.067 

0 

0 

0 

0 

0.143/ 

m 

TABLE  E.1  :  JACCARD  COEHTCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 


TABLE  E.1  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 


12 

1.9 

1.18 

1.11 

1.12 

1.13 

1.14 

1.17 

0 

0 

0.50/0 

0.50/0 

0 

0 

0 

1.18 

0 

0 

0 

0.250^V 

0 

0.111 

1.19 

0 

0.1S4 

0 

0 

0.100 

0.125 

0.067 

IJO 

0.118 

0.048 

0 

0 

0.053 

0.133/in 

0.105 

1J2 

0.0S6 

0.600 

0 

0 

0.071 

0.048 

123 

0 

0/0 

0/0 

0.700 

0 

0 

125 

0.QS0 

0.045 

0 

0 

0 

0.118 

0.109 

126 

0 

0 

0 

0 

0 

0 

O/Il 

121 

0.120 

0.036 

0 

0 

0 

0.143/in 

0.074 

ni 

125 

0 

0/VI 

0 

0 

0.143 

0 

0 
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TABLE  E.1  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  I 
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TABLE  E.2  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 


2.1 

2.2 

23 

23 

23 

2.6 

2.1 

IV 

- 

- 

- 

- 

- 

2Jt 

OflV 

IV 

- 

- 

- 

- 

23 

QflV 

0.091AV 

IV 

- 

- 

- 

2.4 

(VIV 

0.09/IV 

0.30(VIV 

IV 

- 

- 

23 

OflV 

0.067/IV 

0.133/IV 

0.133/IV 

IV 

- 

2.6 

0 

0.091 

0.091 

0.083 

0.063 

V 

2.7 

0.077 

0 

0 

0 

0 

0 

23 

0.077 

0 

0 

0 

0 

0 

2.19 

0 

0 

0 

0 

2.11 

0 

0 

0 

0 

0 

0 

2.12 

0 

0 

0 

0 

0 

0 

2.13 

0 

0.111 

0.091 

0.091 

0.154 

0 

2.14 

0 

0 

0 

0 

0 

0 

2.15 

0 

0 

0 

0 

0 

0 
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TABLE  E.2  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  n 


2.1 

2.2 

2J 

2.4 

IS 

2.6 

2.16 

cwv 

(MV 

(MV 

(MV 

(MV 

0 

2.17 

(VIV 

(MV 

(MV 

(MV 

(MV 

0 

2.18 

(WV 

(MV 

(MV 

(MV 

(MV 

0 

2.19 

(VIV 

(MV 

(MV 

(MV 

(MV 

0 

2.20 

(WV 

(MV 

(MV 

(MV 

(MV 

0 

2.21 

(WV 

(MV 

(MV 

(MV 

(MV 

0 

2.22 

(MV 

(MV 

(MV 

(MV 

(MV 

0 

2.23 

(MV 

(MV 

(MV 

(MV 

(MV 

0 
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TABLE  E.2  :  JACCARD  COEFTICIENTS  OF  PASTURE  REGIONS  OF 

VERSION  n 


■a 

2J 

2.19 

2.11 

2.12 

2.13 

2.14 

2.15 

2.16 

0 

0 

0.400 

0.400 

2.17 

0 

0 

0 

0.400 

0.400 

2.1S 

IQH 

0 

0 

0.400 

0.400 

2.19 

0 

0 

0 

0 

0 

2M 

0 

0 

0 

0 

0 

2.21 

0 

0 

0 

0 

2J2 

0 

0 

0 

0 

0 

0 

0 

0 

2J3 

0 

HHQIIIII 

IHQHI 

0 

mQiin 

mQnni 

2.2S 

0 

0 

0 

0 

0 

0 

0 

2M 

(yii 

(VII 

0 

0 

0 

0 

(VII 

2J$ 

0 

0 

0 

HQH 

0 

0 

0 

0 

2Ji 

0 

HQHH 

mQiiiii 

0 

0 

0 
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TABLE  :  JACCARD  COEFUaENTS  OF  FAILURE  REGIONS  OF 

VERSION  m 


TABLE  E3  :  JACCARD  COEFTIOENTS  OF  FAILURE  REGIONS  OF 

VERSION  m 


3.11 

3.12 

3.13 

3.1S 

3.U 

3.17 

3.1S 

3.19 

3.21 

3.22 

3.2 

3 

3^ 

0 

0 

0 

a  to 

a 

(VO 

0.07 

0 

(VO 

0 

0.1 

/o 

■■ 

7 

82 

3M 

an 

aio 

a 

ao9 

an 

0 

ao8 

0 

0 

0.16 

0.0 

1 

0 

WM 

1 

1 

3 

7 

83 

3M 

n 

D 

D 

ao9 

a 

0/0 

0 

D 

(VO 

0 

0.0 

■■ 

BiB 

■i 

I/O 

■fl 

H 

83 

3M 

an 

an- 

0 

0 

0.11 

II 

0.06 

0 

0 

0.11 

0.0 

i/v 

i/v 

1/v 

H 

3 

1 

83 

332 

0 

D 

0 

0.10 

0 

0/0 

0 

0 

0.33 

0 

0 

■b 

(VO 

3/0 

333 

D 

0 

a 

0.09 

0 

0 

0 

0 

0.25 

0 

0 

■■ 

■B 

1 

0 

/VI 

334 

D 

0 

(VII 

0.09 

0 

D 

0 

0 

0.25 

0 

0 

■■ 

1 

WM 

0 

335 

B 

B 

0 

ao9 

1 

0 

B 

0 

0 

0.25 

0 

0 

TABLE  EJ  :  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 
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TABLE  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  m 


■ 

3.11 

3.12 

3.13 

3.1 

5 

H 

Q 

3.19 

3.18 

331 

3.22 

3.23 

336 

0 

0 

0 

0X791 

/IV 

0 

0 

0 

0 

0.250 

0 

0  4V 

331 

D 

D 

0 

0u091 

/IV 

0 

0 

0 

0 

0.250 

0 

0/IV 

336 

0 

0 

0 

Q 

0 

0 

0 

0.063 

B 

0 

ox)e> 

yiv 

339 

0 

0 

0 

H 

D 

0 

D 

0 

0 

D 

0X70- 

31/IV 

3.46 

D 

D 

a 

m 

0 

0 

0 

0 

0 

0 

0X76- 

3WV 

3.41 

D 

D 

a 

D 

0 

D 

0 

0 

0 

0X76- 

3/IV 

3.42 

0 

0 

a 

M 

0 

0 

0 

0 

0 

0 

0X76- 

3/IV 

3.43 

0.013 

0.077 

(VII 

0 

0 

0 

0.143 

0 

0.111 

0X767 

3.44 

a067 

a063 

0 

0 

0.06 

7 

0 

(Vin 

a9io 

/m 

0 

0.077 

0.056 

3.45 

0 

0.05 

0 

0 

0 

0 

(Vin 

0.471 

m 

0 

0 

0 
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TABLE  E3  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  m 


334 

3J5 

334 

3JS 

3J2 

3J3 

334 

335 

3^4 

0 

- 

- 

- 

- 

- 

• 

- 

3^5 

0 

1 

- 

- 

- 

- 

- 

- 

3M 

0.333 

to 

0 

0 

- 

- 

- 

- 

- 

3M 

0.125 

0.111 

0.333 

V 

- 

- 

- 

- 

332 

(VO 

0.1C7/O 

0 

0 

- 

- 

- 

333 

0 

0 

ai43 

0 

0300 

VI 

- 

- 

334 

0 

ai43 

0300 

a400 

n 

- 

335 

0 

0 

ai43 

0 

0300 

0.400 

IV 

3J6 

0 

0.143 

0 

0.500 

0400 

0.400 

0L4(yiv 

331 

0 

ai43 

0 

0.400 

04(VIV 

3J8 

0.286 

0 

0375 

0300 

0 

0 

0 

(Viv 

3J9 

0.143 

0 

03SO 

0.100 

0 

0 

0 

(WV 

3.4# 

0.143 

0 

aioo 

0 

0 

0 

(VIV 

3.41 

0.143 

0 

0350 

0.100 

0 

0 

(WV 

3.42 

0.143 

0 

0350 

0.100 

0 

0 

0 

(VIV 

3.43 

0 

0.125 

0 

0X>83 

0 

0 

O/II 

0 

3.44 

0.077 

0.083 

0 

0X)63 

0 

0 

0 

0 

3.45 

0.059 

0 

0.059 

0.053 

0 

0 

0 

0 
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TABLE  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  m 


■ 

334 

337 

33$ 

3J9 

3.4t 

3.41 

3.42 

3.43 

3.45 

3J6 

IV 

- 

- 

- 

- 

- 

- 

- 

- 

- 

3J7 

a400 

/IV 

IV 

- 

- 

- 

- 

- 

- 

- 

' 

3J$ 

O/IV 

H 

IV 

- 

- 

- 

- 

- 

- 

- 

339 

OAV 

0 

AV 

028- 

6/IV 

IV 

- 

- 

- 

- 

- 

- 

OAV 

0 

AV 

0286 

/IV 

osoo 

/IV 

IV 

- 

- 

- 

- 

- 

3.41 

m 

0 

AV 

0286 

/IV 

050- 

OflV 

OJOO 

/IV 

IV 

■ 

- 

- 

- 

3A2 

m 

0 

AV 

0286 

/IV 

OSOO 

/IV 

OSOO 

/IV 

0.500 

/IV 

IV 

- 

- 

- 

3.43 

0 

MM 

0 

0091 

0.091 

0.091 

0.091 

MM 

- 

- 

3.44 

0 

0 

0 

0 

0 

0 

■ 

B 

- 

3.45 

0 

a 

OIOS 

0.0S3 

0.053 

0.0S3 

0.0S3 

0X>S3 

(Vin 

B 
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TABLE  E.4  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


VERSION  IV 


■ 

4.1 

4.3 

4.3 

4.4 

4.S 

4.4 

4.7 

4.3 

4.3 

4.13 

4.11 

4.1 

IV 

- 

- 

- 

. 

. 

- 

- 

- 

- 

- 

- 

4.3 

0 

V 

- 

- 

. 

- 

- 

- 

- 

- 

- 

- 

4.3 

0 

aiu 

m 

- 

- 

- 

- 

- 

- 

- 

- 

4.4 

0 

0 

0j045 

n 

- 

- 

- 

- 

- 

. 

- 

- 

4.S 

B 

B 

0D63 

ojm 

/n 

B 

B 

- 

B 

- 

B 

B 

- 

4.4 

B 

B 

0 

B 

E9 

B 

- 

- 

- 

- 

. 

. 

4.7 

B 

B 

0X171 

/n 

ai6- 

7/n 

(MI 

n 

- 

- 

- 

- 

- 

4.3 

■ 

■ 

0.12S 

■ 

am 

/n 

_ 

B 

B 

B 

B 

■ 

4.3 

■ 

■ 

Oi)63 

mm 

0.25- 

(vn 

B 

0.125 

B 

B 

B 

■ 

4.13 

B 

B 

0.056 

0X16- 

3/n 

o/n 

m 

ai25 

/n 

0091 

ai2- 

5ra 

B 

■ 

■ 

4.11 

O/IV 

oxm 

ISI 

0 

B 

0 

0 

0 

B 

0 

IV 

■ 

4.13 

B 

B 

0.063 

0.21- 

4/n 

ai6- 

7/n 

CMI 

al^ 

»n 

aioo 

0.167 

/n 

0.10- 

vn 

B 

D 

4.13 

(WV 

0 

HI 

0 

B 

0 

0 

0 

0 

0 

0 

D 

4.14 

O/IV 

0 

0 

0 

0 

0 

0 

0 

B 

B 

B 

a 

4.15 

0.500 

0 

0.091 

0 

0 

0 

0 

0214 

0 

0 

0 

D 

4.14 

B 

0D71 

B 

ais- 

i/n 

OfU 

B 

B 

B 

B 

123 

4.13 

0 

0.100 

0 

IQIf 

vn 

0 

■fi 

(MI 

0 

(MI 

4.13 

0 

0.100 

mm 

(Ml 

(Ml 

B3 

om 

cm 

0 

B3 

4J3 

B 

B 

0 

^9 

(Ml 

d 

B 

IB 

om 

B 

IQ] 

TABLE  E.4  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  IV 


4.13 

4.14 

4.15 

4.16 

4.18 

4.19 

4.20 

4J 

1 

Q 

4J4 

4.13 

IV 

- 

- 

- 

- 

- 

- 

- 

- 

— 

■ 

4.14 

0.800 

/IV 

IV 

- 

- 

> 

- 

- 

- 

- 

- 

- 

4.15 

(VIV 

(VIV 

IV 

- 

- 

■ 

■ 

- 

- 

■ 

■ 

4.16 

0 

0 

0 

n 

- 

■ 

- 

- 

- 

■ 

■ 

4.18 

0 

0 

0 

0.09- 

m 

D 

- 

- 

- 

- 

- 

- 

4.19 

0 

0 

0 

0.07- 

1/n 

0.12- 

vn 

- 

- 

- 

- 

- 

4.20 

0 

0 

0 

(WI 

(WI 

a 

_ 

4J1 

0 

0 

0.063 

0i)83 

0.077 

0 

VI 

- 

- 

B 

4J2 

0 

0 

0 

(WI 

(WI 

(WI 

(WI 

0 

D 

- 

B 

4.24 

Q/IV 

(VIV 

(VIV 

C.083 

0.167 

0.143 

0 

0 

0 

IV 

B 

4J6 

(VIV 

(WV 

(WV 

0.071 

0.125 

0.111 

0 

0.077 

0 

aoo 

JIV 

TABLE  E.4  :  JACCARD  COEFnCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  IV 


4.1 

4.2 

D 

D 

4.S 

4.6 

4.7 

4.8 

4.9 

4.19 

4.11 

4.21 

0 

0.071 

0 

B 

0 

B 

0 

0 

0 

0 

0 

4J2 

0 

0 

0 

(vn 

(vn 

m 

B 

(vn 

^9 

4J4 

0/IV 

0.100 

0 

0 

0 

0 

0 

0 

0 

(VIV 

D 

4J6 

0/IV 

0.083 

mm 

B 

0 

0 

0 

0 

0 

0 

(V.IV 

B 

TABLE  E.5:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  V 


1 

la 

5J 

5J 

5.4 

5S 

mm 

E9 

5J 

mm 

n 

- 

- 

- 

- 

- 

- 

- 

5J 

IV 

- 

- 

- 

- 

- 

- 

0 

O 

- 

- 

- 

- 

- 

5^ 

0 

aos3/lv 

IV 

- 

- 

- 

- 

5^ 

0 

OjQSWIV 

0 

0375/IV 

IV 

- 

- 

- 

mm 

0.167/0 

0 

o 

- 

- 

- 

0 

0 

0/0 

0 

0 

0/0 

0 

- 

- 

5^ 

0 

0.111/IV 

0 

0.0SWIV 

a067/IV 

0 

0 

rv 

- 

m 

OjOS3 

Q/O 

0 

0/0 

0/0 

0 

0 

5.1« 

0 

0 

(VO 

0 

0 

0/0 

0/0 

0 

0.143 

10 

5.11 

0 

0 

(VO 

0 

0 

0/0 

0/0 

0 

0/0 

5.12 

0 

(VO 

0 

0/0 

0/0 

0 

0/0 

5.13 

0 

0.050 

0 

0.083 

0.100 

0 

0.056 

5.14 

0 

0 

0/0 

0 

0 

0/0 

0/0 

IIIQ^ 

0/0 

5.15 

0 

0 

0 

0 

0 

0 

0 

0 

0 

5.16 

jj^QIII 

0 

0 

0 

0 

0 

0 

0 

5.17 

(VII 

0 

0 

0 

0 

0 

^QUI 

0 

5.18 

0 

0 

0 

0 

0 

0 

0 

0 

5.19 

[H^H 

(WV 

0 

(WV 

(Viv 

0 

0 

(WV 

0 
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TABLE  E.5:  JACCARD  COEFUCIENTS  OF  FAH^URE  REGIONS  OF 

VERSION  V 


157 


TABLE  E^:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  V 


S.lf 

SM 

5J1 

SM 

5J3 

5J4 

5J5 

5M 

5J7 

5J1 


5M  WW  0.286  (VIV 


5M  0.44WV 


0.44VIV  CWV  IV 


5J4  (VIV  0.286  (VIV  (yiv 


5M  I  0.44VIV 


0.444fIV  GWV  0L444/IV  Qfiy  jy 


5M  (VIV  0.286  (VIV  (yiv 


5^7  0l44</IV  0 


5^ 


(WV 

0 

(VIV 

(VIV 

(VIV 

(VIV 

(WV 

(WV 

(WV 

(WV 

0 

(VIV 

(VIV 

(VIV 

(VIV 

(WV 

(WV 

(WV 

0 

m 

0 

0 

0 

0 

0 

0 

0 

S.42 

0 

IIIIIIQ^ 

0 

0 

IIIIQ^ 

0 

0 

0 

0 

5.43 

0 

(VII 

0 

0 

0 

0 

0 

0 

0 
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TABLE  E.5:  JACCARD  COEFnClENTS  OF  FAILURE  REGIONS  OF 

VERSION  V 


5J8 

5J# 

5J1 

5.32 

5J3 

5J4 

5J 

5 

53 

7 

5J 

9 

H 

B 

5.4 

2 

ISI 

m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

isi 

(VO 

0 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 
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(VO 

(VO 

El 

- 

- 

- 

- 

- 

- 

- 

- 

- 

ISI 

(VO 

(VO 

(VO 

0 

- 

- 

- 

- 

- 

- 

- 

- 

0 

D 

D 

D 

IV 

- 

- 

- 

- 

- 

- 

- 

g 

D 

0 

0 

o 

(VIV 

IV 

m 

■ 

■ 

■ 

■ 

■ 

■ 

5J5 

0 

B 

OlOS3 

0 

0 

0214 

VI 

B 

B 

B 

B 

B 

5J7 

a 

D 

D 

0 

a 

0 

a 

- 

■1 

- 

5J9 

0 

0 

0.12S 

0 

m 

H 

0 

IV 

■ 

B 

■ 

5.4# 

0 

0 

0 

B 

Ofiv 

(VI 

V 

0 

B 

OjC»- 

3/IV 

IV 

B 

■ 

5.41 

D 

a 

0 

0 

0 

(VII 

D 

(VII 

0 

0 

n/y 

■ 

5.42 

(VO 

(VO 

(VO 

(VO 

a 

0 

0 

0 

0 

0 

0 

5.43 

0 

0 

0 

HI 

a 

(VII 

0 

■HBVl 

(VII 

0 

0 

(VII 

m 
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TABLE  E,i  :  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


TABLE  E.6  :  JACCARD  COEfTIOENTS  OF  FAILURE  REGIONS  OF 

VERSION  VI 


TABLE  E.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS 


OF  VERSION  Vn 


mam 

74 

74 

74 

74 

74 

WSM 

■ai 

n 

- 

- 

- 

- 

- 

- 

7JS 

0.083 

in 

- 

- 

- 

- 

- 

7J 

a083 

0.818/in 

- 

- 

- 

- 

7.4 

IV 

- 

- 

- 

7.S 

0.167/11 

0 

0 

0.071 

n 

- 

- 

mm 

0 

0 

0 

0 

0 

0 

- 

7.7 

0 

0 

0.333/0 

3{3^| 

74 

0 

0 

0.333A3 

0.S0Q/O  1 

74 

0 

0.333^V 

13^31 

3331 

13133 

7.1t 

(Vll 

0.231 

0.214 

0 

(Vll 

1331 

0 

7.11 

n[iiQm 

0 

0 

0 

0 

a2oo 

0.167 

7.12 

iinQiiiiii 

1111111^311 

HHQHIII 

0 

0 

0 

0 

7.13 

(Vll 

13311 

0 

(Vll 

310^1 

0 

7.14 

0 

0 

3133 

0.143 

0 

0/0 

0/0 

7.15 

1^31 

ao9i/in 

0.083/in 

0 

0 

0 

0 

7.14 

0 

0 

0 

0 

0 

0 

0 
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TABLE  E.7:  JACCARD  COEFUCIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


7.1 

73 

73 

7.4 

73 

7.6 

7.17 

(VII 

0 

0 

0 

MI 

0 

7.IS 

0 

0 

(VIV 

0 

0 

7.1» 

0 

0 

0 

MV 

0 

IM 

0 

0 

0 

MV 

0 

0 

7.J1 

0 

0 

0 

MV 

0 

0 

7J2 

0 

0 

MV 

0 

0 

7J3 

0 

0 

MV 

0 

0 

7J4 

0 

0 

0 

MV 

0 

0 

7JS 

0 

0 

MV 

0 

myin 

7.37 

aioo 

o.20(vin 

0.273/in 

0 

0 

0 

IM 

0 

0 

0 

MV 

0 

0/0 

7J9 

asm 

a45SAn 

0.417/in 

0 

0 

0 

7J2 

o.mni 

0.038 

0.037 

0 

MI 

0 

7J3 

0 

0 

0 

0 

0 

0.143 

735 

0.077/11 

0.333 

0.400 

0 

MI 

0 

TABLE  E.7:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 

VERSION  vn 


D 

7J 

D 

7.18 

7.11 

7.12 

7.13 

H 

7.15 

7.16 

7.17 

0 

(WI 

■■ 

0 

(WI 

D 

0 

0.400 

7.18 

0 

0 

(Viv 

0 

0 

■a 

0 

D 

0 

0.400 

7.19 

(WV 

0 

a 

0 

0 

0 

0 

0.400 

7J8 

0 

0 

(WV 

0 

0 

0 

0 

0 

0 

0.400 

7J1 

0 

(VIV 

0 

0 

0 

D 

0 

0 

0 

7J2 

(WV 

n 

0 

0 

a 

0 

0 

0 

7J3 

0 

(VIV 

0 

0 

0 

0 

0 

0 

0 

7J4 

(WV 

0 

0 

0 

0 

0 

s 

0 

7J5 

0 

0 

(WV 

0 

0 

D 

0 

0 

B 

0 

7J7 

0 

0 

0 

0077 

0 

D 

0 

0 

oiii/m 

0 

7J8 

(VO 

(VO 

(WV 

0 

0 

0 

0 

(VO 

0 

0 

7J9 

0 

0 

0 

0.231 

0 

0 

0 

0 

0.083 

0 

7J2 

0 

0 

(WI 

0 

0 

(WI 

0 

oxtso 

0 

7J3 

0.125 

0.125 

0 

0 

0333 

/V 

0 

0 

0 

0 

0 

7J5 

0 

D 

0 

0J20- 

(vn 

0 

0 

(WI 

0 

0.091 

0 
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7.17 

7.1t 

7.19 

7M 

7.21 

7.17 

n 

- 

- 

- 

- 

7.1t 

0.400 

IV 

- 

- 

- 

7.1> 

a4oo 

0.40(VIV 

IV 

- 

- 

IM 

a400 

0.40(VIV 

0.40(VIV 

IV 

- 

7J1 

0 

OfiV 

(WV 

(VIV 

IV 

7.22 

0 

(Viv 

(VIV 

(VIV 

0.571/IV 

7.23 

0 

(Viv 

(VIV 

(VIV 

0.571/IV 

7J4 

0 

(VIV 

(VIV 

(VIV 

0.571/IV 

7J5 

0 

(VIV 

(VIV 

0/IV 

0.429/IV 

7J7 

0 

0 

0 

0 

0 

IM 

0 

(VIV 

(VIV 

(VIV 

(VIV 

7M 

0 

0 

0 

0 

0 

7J2 

(VII 

0 

0 

0 

0 

7J3 

0 

0 

0 

0 

0 

7J5 

(VII 

0 

0 

0 

0 
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122 

123 

124 

125 

121 

7.28 

7.29 

7.32 

IQI 

7J5 

121 

IV 

- 

- 

- 

- 

- 

- 

- 

- 

- 

121 

0.571 

AV 

IV 

- 

- 

- 

- 

- 

- 

- 

- 

124 

0.571 

AV 

0.571 

AV 

IV 

- 

- 

- 

- 

- 

- 

- 

12S 

0.429 

AV 

0.429 

AV 

0.429 

AV 

IV 

- 

- 

- 

- 

- 

- 

121 

0 

0 

0 

0 

- 

- 

- 

- 

- 

12S 

OAV 

CAV 

OAV 

OAV 

0 

O/IV 

- 

- 

- 

- 

129 

0 

0 

0 

0 

0.182 

An 

0 

D 

- 

- 

- 

122 

0 

0 

0 

0 

0.125 

0 

0.037 

D 

123 

0 

0 

0 

0 

0 

0 

0 

0 

D 

- 

125 

0 

0 

0 

0 

0.200 

0 

0333  i  0.034 

1  /n 

0 

D 

TABLE  E.8:  JACCARD  COEFFICIENTS  OF  FAILURE  REGIONS  OF 


VERSION  vm 


8.1 

8.2 

8J 

8.4 

8J 

8.6 

8.7 

8.8 

8.1 

- 

- 

- 

- 

- 

- 

- 

SJZ 

0 

I 

- 

- 

- 

- 

- 

- 

0 

0.500 

IV 

- 

- 

- 

- 

- 

D 

0 

0.143 

0.2S0 

/IV 

IV 

- 

- 

- 

- 

8.5 

0 

1.00 

osoo 

/IV 

0.429 

/IV 

IV 

- 

- 

- 

8.6 

0 

0.500 

(Viv 

0.12S 

/IV 

0.500 

AV 

IV 

- 

- 

8.7 

0 

0.333 

OMT/fV 

0.111  /IV 

0l333/IV 

0.667 /IV 

IV 

- 

8A 

0^ 

0 

0 

0 

0 

0 

V 

8.9 

0 

0 

0 

0 

0 

0 

0 

0 

8.18 

0 

0 

0 

0 

0 

0 

0 

0/V 

8.11 

0 

0.333 

0.200 

0.100 

0.250 

0.200 

0.167 

0.667 

/V 

8.12 

0 

0 

(WV 

OAV 

OAV 

OAV 

0 

0 

8.13 

0.730 /m 

0 

0 

0 

0 

0 

0 

0.500 

8.14 

0.222 

0 

0 

0 

0 

0 

0 

8.15 

0.750 

0 

0 

0 

0 

0 

0 

1.00 

8.18 

0.143 

on 

0 

0 

0 

0 

0 

0 

8.19 

0.286 

0 

0 

0 

0 

0 

0 

0.500 

8J3 

0 

0.500/1 

0.333 

0.111 

0.333 

0333 

0.250 

1.00 

8.25 

0 

0 

0 

0 

0 

0 

0 

0 

8.26 

0 

0 

0 

0 

0 

0 

0 

0 
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■ 

8.9 

8.10 

8.11 

8.12 

8.13 

8.14 

8.1 

a 

Bl 

■ 

5 

D 

n 

3 

5 

D, 

D 

- 

- 

- 

m 

- 

- 

- 

- 

- 

- 

- 

8.1* 

0 

V 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

8.11 

0 

(W 

V 

- 

- 

- 

- 

- 

- 

- 

- 

- 

8.12 

0 

0 

0 

IV 

- 

- 

- 

- 

- 

- 

- 

- 

8.13 

0.16 

0 

0 

D 

o 

■■ 

■ 

o 

H 

• 

• 

. 

7 

■■ 

m 

■ 

■ 

■ 

8.14 

0.09 

D 

D 

D 

0.12 

a 

H 

■ 

• 

• 

l/II 

■I 

■I 

■I 

5 

■■ 

■ 

■ 

8.15 

0.20 

0 

0 

0 

0.429 

D 

■ 

■ 

AI 

mm 

■ 

■ 

■ 

8.18 

0 

0 

0 

0 

0 

0.100 

1 

- 

- 

- 

- 

8.19 

0.12S 

0.1«7 

D 

0.400 

0.2 

0.75 

0 

a 

■■ 

H 

■1 

mm 

H 

8.23 

0 

0.50 

0.50 

0 

0 

■ 

■ 

O/I 

■ 

I 

- 

- 

8J5 

0 

0 

0 

0 

0 

a 

n 

0 

0 

0 

VI 

- 

8J6 

(VII 

0 

0 

0 

0 

O/Q 

il 

0 

0 

0.6«7 

a 
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D 

8.2 

8J 

la 

8J 

8.6 

8.7 

8.8 

la 

8.18 

8.27 

0 

0 

(VIV 

(WV 

(WV 

(WV 

(WV 

0 

IIQH 

0 

0 

0 

(WV 

(WV 

(WV 

(WV 

(WV 

0 

0 

0 

8.29 

■■ 

0 

(VIV 

dV 

(WV 

(WV 

(WV 

0 

0 

a 

8.39 

D 

(VIV 

(WV 

(WV 

(WV 

(WV 

0 

0 

0 

8J1 

0 

0 

(WV 

(WV 

(WV 

(WV 

(WV 

0 

0 

0 

8.32 

0 

D 

(WV 

(WV 

(WV 

(WV 

(WV 

0 

0 

0 

8J3 

0 

0 

(WV 

(WV 

(WV 

(WV 

(WV 

0 

0 

0 

8J4 

0 

0 

(WV 

(WV 

(WV 

(WV 

(WV 

0 

0 

8J5 

■a 

■■ 

0 

0 

■a 

0 

0 

0 

0 

8J6 

0.333 

0 

D 

a 

0 

0 

0.143 

0.091 

ftt 

0 

8J7  a600  0 


8J8  0.600  0 


8J9  0 


8.48  0.S00  0 


8.41  0.500  0 


8.44  0.333  0 


0  0.333  I  0.143 


0.333  0.143 

/n 


(VII 


0.250  I  0.125 


0-333  0.143  0 

ni 


0333  (W 


0 


8.45  0.200  1.00  0.333  0.125 


8.46  1^333  I  0  I  0  I  0 


0300  0333 


0 


0.167  0.100  O/V 

/V 


0.250  Ofy 


0.333  0333  0.200  O/V 

/V 


8.48 

o.s/in 

0 

0.500 

0.143 

1.00 

0300 

0 

0 

0 

a 

8.58 

0 

0 

(WV 

(WV 

(WV 

(WV 

0/IV 

0 

0 

0 
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